Methods and Systems for Generating Depth Data by Converging Independently-Captured Depth Maps

ABSTRACT

An exemplary depth data generation system (“system”) accesses a first depth map and a second depth map of surfaces of objects included in a real-world scene. The first and second depth maps are captured independently from one another. The system converges the first and second depth maps into a converged depth map of the surfaces of the objects included in the real-world scene. More specifically, the converging comprises assigning a first confidence value to a first depth data point in the first depth map, assigning a second confidence value to a second depth data point in the second depth map, and generating a third depth data point representing a same particular physical point as the first and second depth data points based on the first and second confidence values and on at least one of the first depth data point and the second depth data point.

BACKGROUND INFORMATION

Depth data (e.g., spatial location data, positional coordinate data,etc.) representative of surfaces of objects in the world may be usefulin various applications. For example, depth data representative ofobjects in a real-world scene may be used to generate virtual realitycontent that includes an immersive virtual reality world that mimics thereal-world scene. Accordingly, users (e.g., people using the virtualreality content by way of a media player device) may virtuallyexperience the real-world scene by viewing and/or interacting with anyof a variety of things being presented in the immersive virtual realityworld.

Current techniques for capturing depth data may have room forimprovement, particularly in the context of capturing depth datarepresentative of objects included in a real-world scene in virtualreality applications. For example, while various fixed positions withrespect to the real-world scene (e.g., various perspectives, angles,vantage points, etc., on the real-world scene) and/or various differentdepth capture techniques may potentially be available for capturingdepth data representative of objects in the real-world scene, a fixedposition and/or a depth capture technique that may be ideal forcapturing depth data representative of a particular object or surface ofan object in the real-world scene may be different from a fixed positionand/or a depth capture technique that would be ideal for capturing depthdata representative of a different particular object or another surfaceof the object in the real-world scene. Accordingly, regardless of whichdepth capture technique and/or which fixed position is used to capturedepth data for a particular real-world scene, depth data captured torepresent at least some objects and/or surfaces of objects in thereal-world scene may be inaccurate, imprecise, suboptimal, deficient, orotherwise leave room for improvement.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate various embodiments and are a partof the specification. The illustrated embodiments are merely examplesand do not limit the scope of the disclosure. Throughout the drawings,identical or similar reference numbers designate identical or similarelements.

FIG. 1 illustrates an exemplary depth data generation system forgenerating depth data by converging independently-captured depth mapsaccording to principles described herein.

FIG. 2A illustrates an exemplary configuration in which depth datarepresentative of exemplary surfaces of an object in a real-world sceneis captured by an exemplary depth map capture subsystem according toprinciples described herein.

FIG. 2B illustrates a perspective view of the object of FIG. 2A from theperspective of the depth map capture subsystem of FIG. 2A according toprinciples described herein.

FIG. 3 illustrates an exemplary implementation of the depth datageneration system of FIG. 1 positioned with respect to an exemplaryreal-world scene in order to generate depth data by convergingindependently-captured depth maps according to principles describedherein.

FIG. 4 illustrates an exemplary node of an exemplary implementation ofthe depth data generation system of FIG. 1 according to principlesdescribed herein.

FIG. 5 illustrates another exemplary implementation of the depth datageneration system of FIG. 1 positioned with respect to another exemplaryreal-world scene in order to generate depth data by convergingindependently-captured depth maps according to principles describedherein.

FIG. 6 illustrates an exemplary virtual reality experience in which auser is presented with exemplary virtual reality media contentrepresentative of a real-world scene as experienced from a dynamicallyselectable viewpoint corresponding to an exemplary arbitrary locationwithin the real-world scene according to principles described herein.

FIGS. 7A-7C illustrate exemplary depth map capture techniques accordingto principles described herein.

FIG. 8A illustrates exemplary components of another exemplaryimplementation of the depth data generation system of FIG. 1 generatingdepth data by converging independently-captured depth maps according toprinciples described herein.

FIGS. 8B-8C illustrate perspective views of the object of FIG. 8A fromthe perspective of the depth map capture subsystems of FIG. 8A accordingto principles described herein.

FIG. 9 illustrates an exemplary converging of two independently-captureddepth maps into exemplary converged depth maps according to principlesdescribed herein.

FIGS. 10-11 illustrate exemplary methods for generating depth data byconverging independently-captured depth maps according to principlesdescribed herein.

FIG. 12 illustrates an exemplary computing device according toprinciples described herein.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Methods and systems for generating depth data by convergingindependently-captured depth maps are described herein. As used herein,“depth data” may broadly include any spatial location data, positionalcoordinate data, or other data representative of a position of one ormore surfaces (e.g., or, more particularly, one or more physical pointson the surfaces) of one or more objects in three-dimensional (“3D”)space. For example, as will be described and illustrated below, depthdata may include data representative of surfaces of objects included ina real-world scene. Depth data may be captured in various ways and/or byway of various techniques including by methods and systems describedherein. In certain examples, depth data may be combined and/orsynchronized with video data (e.g., two-dimensional (“2D”) video data)to generate a dynamic volumetric model of the surfaces of objects thatincorporates the depth data and the video data over a period of time.Such volumetric models may be used to generate virtual reality contentsuch as, for example, virtual reality content including an immersivevirtual reality world representative of a real-world scene that includesthe objects. Examples of depth data, techniques for capturing depthdata, and uses for depth data are described herein.

In some examples, depth data may be captured, generated, stored,transmitted, or otherwise created or processed in the form of or as partof a “depth map.” As used herein, a depth map may be representative ofat least one surface of an object (e.g., an object included within areal-world scene) by including or implementing depth data (e.g., depthdata points each representative of a particular physical point on asurface of an object) that describes the spatial location, positionalcoordinates, etc., for the surface of the object. For example, a depthmap may represent particular surfaces of various objects included withina real-world scene, where the particular surfaces are the surfaces ofthe objects that may be seen from one particular fixed position orperspective. Thus, a depth map may not include all the depth data thatmay be used to fully model all the surfaces of an object (e.g., all thesurfaces of the object that may be observed from all vantage pointsaround the object), but, rather, may include a portion of the depth datathat may be used to fully model all the surfaces of the object.

In certain examples, depth maps may be captured from different fixedpositions (e.g., different vantage points or perspectives) with respectto the real-world scene, by way of different depth map capturetechniques (e.g., different technologies for capturing depth data,examples of which will be provided below), and/or by different depth mapcapture subsystems. Such depth maps may be referred to herein as being“independently captured” if one depth map does not derive from orotherwise rely on the other depth map. As will be described below inmore detail, multiple independently-captured depth maps may be converged(e.g., merged, combined, etc.) to form converged depth maps that aremore accurate, more comprehensive (e.g., covering more perspectives),and/or otherwise superior to the independently-captured, non-convergeddepth maps.

In order to generate depth data by converging independently-captureddepth maps, a depth data generation system may access a first depth mapof surfaces of objects included in a real-world scene and a second depthmap of the surfaces of the objects included in the real-world scene,where the second depth map is captured independently from the firstdepth map (e.g., captured from a different fixed position, by way of adifferent depth map capture technique, by a different depth map capturesubsystem, etc.). The first depth map may include a first plurality ofdepth data points each representative of a different physical pointincluded in a plurality of physical points on the surfaces of theobjects included in the real-world scene. Similarly, the second depthmap may include a second plurality of depth data points eachrepresentative of a different physical point included in the pluralityof physical points on the surfaces of the objects included in thereal-world scene. In other words, the first and second pluralities ofdepth data points included within the first and second depth maps,respectively, may represent the same plurality of physical points on thesurfaces of the objects included in the real-world scene asindependently captured from different fixed positions and/or by way ofdifferent depth map capture techniques.

As referred to herein, a “plurality of physical points” on the surfacesof the objects may refer to physical points that may be detected fromperspectives of at least two fixed positions at which depth map capturesubsystems are positioned to capture depth data. In other words, bydefinition, each physical point in the plurality of physical points maybe represented by at least one depth data point in the first pluralityof depth data points included in the first depth map and one depth datapoint in the second plurality of depth data points included in thesecond depth map. However, it will be understood that, due tocircumstances associated with each independent capturing of the depthdata points (e.g., the perspectives from which the depth data points arecaptured, etc.), various physical points on the surfaces of the objectsmay not be represented by any depth data point in the first or seconddepth data map (e.g., physical points on a surface that is not facingthe fixed positions from which the first and second depth maps arecaptured). Additionally, and for the same or similar reasons, it will beunderstood that the first depth map may include certain depth datapoints that represent physical points on the surfaces of the object thatare not represented by corresponding depth data points included in thesecond depth map, and vice versa.

The depth data generation system may converge the first and second depthmaps into a converged depth map of the surfaces of the objects includedin the real-world scene. For example, the converged depth map mayinclude a third plurality of depth data points each representative of adifferent physical point included in the plurality of physical points onthe surfaces of the objects included in the real-world scene (i.e., thesame plurality of physical points represented in both the first and thesecond depth maps). The converging may include assigning a firstconfidence value to a first depth data point in the first plurality ofdepth data points of the first depth map, and assigning a secondconfidence value to a second depth data point in the second plurality ofdepth data points of the second depth map. The first depth data pointand the second depth data point may each represent a same particularphysical point included in the plurality of physical points on thesurfaces of the objects included in the real-world scene. Then, theconverging may include generating, based on the first and secondconfidence values and on at least one of the first depth data point andthe second depth data point, a third depth data point (e.g., in thethird plurality of depth data points) representing the particularphysical point. Examples of generating depth data representative of thesurface of the objects by converging independently-captured depth maps,as well as uses for the generated depth data, will be described in moredetail below.

As used herein, a “real-world scene” may refer to any real-worldscenery, real-world location, real-world event (e.g., live event, etc.),or other subject existing in the real world (e.g., as opposed toexisting only in a virtual world) as may serve a particularimplementation. For example, the real-world scene may include any indooror outdoor real-world location such as the streets of a city, a museum,a scenic landscape, a satellite orbiting and looking down upon theEarth, the surface of another planet, or the like. In certain examples,the real-world scene may be associated with a real-world event such as asporting event (e.g., a basketball game, an Olympic event, etc.), aconcert (e.g., a rock concert in a large venue, a classical chamberconcert in an intimate venue, etc.), a theatrical presentation (e.g., aBroadway musical, an outdoor pageant, etc.), a large-scale celebration(e.g., New Year's Eve on Times Square, Mardis Gras, etc.), a race (e.g.,a stock-car race, a horse race, etc.), a political event (e.g., apresidential debate, a political convention, etc.), or any otherreal-world event that may interest potential users. In the same or otherexamples, the real-world scene may be associated with a setting for afictionalized scene (e.g., a set of a live-action virtual realitytelevision show or movie) and/or any other scene at any other indoor oroutdoor real-world location as may serve a particular implementation.

Accordingly, as used herein, an “object” included in a real-world scene,may include anything, whether living or inanimate, that is associatedwith the real-world scene (e.g., located within or around the real-worldscene) and that is visible from a particular viewpoint with respect tothe real-world scene. For example, if the real-world scene includes areal-world event such as a basketball game, objects for which depth dataof the object surfaces may be captured may include the basketball beingused for the game, the basketball court, the basketball standards (i.e.,the backboards, rims, nets, etc.), the players and refereesparticipating in the game, and/or other objects present at and/orassociated with the basketball game.

By converging independently-captured depth maps to generate depth datain accordance with methods and systems described herein, a depth datageneration system may provide and/or benefit from various advantages.For example, a depth data generation system described herein maygenerate more accurate, precise, optimal, and/or comprehensive depthmaps than may be possible using conventional systems for capturing depthdata. More specifically, conventional systems for capturing depth datamay be constrained to represent each physical point on a surface of anobject based on a single depth map capture technique performed from asingle fixed position. In contrast, as will be described in more detailbelow, depth data generation methods and systems described herein maygenerate depth data that incorporates, within one converged depth map,depth data captured from various fixed positions and/or captured by wayof various depth map capture techniques so that each depth data pointincluded in the converged depth map may be optimized (e.g., moreaccurate, more precise, etc.) for each physical point on the surface ofthe object. In other words, the methods and systems described hereinallow the benefits of a variety of perspectives and depth map capturetechniques to be used and represented in a single depth map so as toavoid potential drawbacks of any single perspective or depth map capturetechnique in capturing the depth data points for a depth map.

Moreover, by facilitating the generation of improved depth data (e.g.,more accurate, precise, and/or comprehensive depth maps), the systemsand methods described herein may also facilitate improved modeling basedon the depth data. For example, by using optimized depth data, a depthdata generation system or another system associated with the depth datageneration system may generate a dynamic volumetric model of thesurfaces of objects in the real-world scene more accurately than waspossible previously. As a result, virtual reality media contentrepresentative of the real-world scene generated based on the dynamicvolumetric model of the surfaces of the objects may also be improved.

One or more of these advantages may ultimately benefit an end user ofthe depth data (e.g., a user experiencing an immersive virtual realityworld generated based on the generated depth data) by providing a higherquality (e.g., more realistic) experience to the end user. For example,in applications involving virtual reality media content representativeof a volumetric model of the real-world scene, the user may becomeimmersed in the real-world scene to an extent that may not be possiblefor people presented with the real-world scene by way of traditionalmedia (e.g., television) or traditional virtual reality media.Volumetric models of the real-world scene may allow users to dynamicallyand arbitrarily move their viewpoint within the real-world event, thusproviding the users with an experience of the real-world event not evenavailable to people physically present at the real-world scene (e.g.,people attending a real-world event). For example, users may be able toexperience a live basketball game as if running up and down the courtwith the players, or experience a live concert as if standing on stagenext to the performers.

Various embodiments will now be described in more detail with referenceto the figures. The disclosed methods and systems may provide one ormore of the benefits mentioned above and/or various additional and/oralternative benefits that will be made apparent herein.

FIG. 1 illustrates an exemplary depth data generation system 100(“system 100”) for generating depth data by convergingindependently-captured depth maps. As shown, system 100 may include,without limitation, a depth map access facility 102, a depth mapconvergence facility 104, and a storage facility 106 selectively andcommunicatively coupled to one another. It will be recognized thatalthough facilities 102 through 106 are shown to be separate facilitiesin FIG. 1, facilities 102 through 106 may be combined into fewerfacilities, such as into a single facility, or divided into morefacilities as may serve a particular implementation. Each of facilities102 through 106 may be distributed between multiple devices and/ormultiple locations as may serve a particular implementation. Each offacilities 102 through 106 will now be described in more detail.

Depth map access facility 102 may include any hardware and/or software(e.g., computing systems and devices, computing software, depth capturedevices, etc.) configured to access one or more independently-captureddepth maps that each include a respective plurality of depth data points(e.g., depth data points each representative of a different physicalpoint included in a plurality of physical points on the surfaces ofobjects included in a real-world scene). Depth map access facility 102may access a depth map in any way described herein and/or as may serve aparticular implementation. For example, in some implementations, depthmap access facility 102 may access a depth map by receiving the depthmap (e.g., from another device or system that captures and transmits thedepth map to system 100), by loading the depth map from a location wherethe depth map is stored (e.g., from depth data 108 in storage facility106), or by otherwise accessing data representative of the depth mapafter the depth map has been captured (e.g., by system 100 or by anothersystem). In the same or other implementations, depth map access facility102 may access the depth map by directly capturing the depth map. Inother words, depth map access facility 102 may include depth capturedevices that may be configured to scan, detect, analyze, determine, andotherwise capture (e.g., by way of a particular depth map capturetechnique) the depth map of the real-world scene directly from theobjects included in the real-world scene.

For example, in certain implementations, the accessing of a first depthmap may include capturing the first depth map by way of a first depthmap capture technique, and the accessing of a second depth map mayinclude capturing the second depth map by way of a second depth mapcapture technique that is different from the first depth map capturetechnique. For example, as will be described in more detail below, twodifferent depth map capture techniques may be performed from similar orthe same fixed positions to generate independently-captured depth mapsthat complement one another due to different strengths characterizingthe different depth map capture techniques.

In other implementations, the accessing of the first depth map mayinclude capturing the first depth map by way of a particular depth mapcapture technique, and the accessing of the second depth map may includecapturing the second depth map by way of the same depth map capturetechnique. For example, as will be described in more detail below, thesame depth map capture technique may be performed from two differentfixed positions to generate independently-captured depth maps thatcomplement one another due to different vantage points (i.e., differentperspectives or angles from which depth data for various surfaces ofvarious objects within the real-world scene may be captured) that areprovided by the different fixed positions from which the depth mapcapture techniques are performed.

In yet other implementations, the accessing of the first and seconddepth maps may include capturing the first and second depth maps by wayof different depth map capture techniques performed from different fixedpositions, or by way of the same depth map capture technique performedfrom the same or similar fixed positions, as may serve a particularimplementation. Examples of accessing depth maps (including by capturingthe depth maps) will be described in more detail below.

Depth map convergence facility 104 may include any suitable hardwareand/or software (e.g., computing systems and devices, computingsoftware, etc.) configured to converge independently-captured depth mapsof surfaces of objects included in a real-world scene into a convergeddepth map of the surfaces of the objects. For example, based on firstand second depth maps that include, respectively, first and secondpluralities of depth data points each representative of differentphysical points included in a plurality of physical points on thesurfaces of the objects, depth map convergence facility 104 may form aconverged depth map that includes a third plurality of depth data pointseach representative of the different physical points included in theplurality of physical points represented by the first and secondpluralities of depth data points.

Depth map convergence facility 104 may converge independently-captureddepth maps in any way described herein and/or as may serve a particularimplementation. For example, referring to the example above where depthmap convergence facility 104 converges the first and second depth mapsinto the converged depth map, depth map convergence facility 104 mayassign a first confidence value to a first depth data point in the firstplurality of depth data points of the first depth map, the first depthdata point representing a particular physical point included in theplurality of physical points. Depth map convergence facility 104 maysimilarly assign a second confidence value to a second depth data pointin the second plurality of depth data points of the second depth map,the second depth data point also representing the particular physicalpoint.

As used herein, “confidence values” may be assigned by system 100 (e.g.,by depth map convergence facility 104) to depth data points that areaccessed (e.g., captured, loaded, etc.) by system 100 to indicate howlikely each depth data point is to accurately reflect the reality (e.g.,the actual position, the actual depth with respect to an element of adepth map capture subsystem performing the detection, etc.) of thephysical point to which the depth data point corresponds. Confidencevalues may be numerical values (e.g., percentage values, numbers on aparticular scale, etc.), binary pass/fail-type values, or any other typeof value as may serve a particular implementation. For example, ifsystem 100 determines that a particular depth data point may berelatively likely to accurately reflect an actual depth of a physicalpoint on a surface of an object (e.g., an actual position of thephysical point in 3D space, an actual depth or distance of the physicalpoint from the element of the depth map capture subsystem performing thedetection, etc.), system 100 may assign that particular depth data pointa relatively high confidence value (e.g., an 80% value, a PASS value,etc.). Conversely, if system 100 determines that a particular depth datapoint may be relatively less likely to accurately reflect the actualdepth of the physical point (e.g., to represent a relatively “roughestimate” of the actual position or depth), system 100 may assign thatparticular depth data point a relatively low confidence value (e.g., a20% value, a FAIL value, etc.). Examples of confidence values and howsystem 100 determines and assigns confidence values will be described inmore detail below.

After assigning the respective first and second confidence values to therespective first and second depth data points, depth map convergencefacility 104 may generate, based on the first and second confidencevalues and on at least one of the first depth data point and the seconddepth data point, a third depth data point (e.g., to be part of thethird plurality of depth data points) that, like the first and seconddepth data points, also represents the particular physical point on theparticular surface of the object. For example, as will be described inmore detail below, depth map convergence facility 104 may generate thethird depth data point by relying exclusively or more heavily onwhichever of the first and second depth data points is more likely to beaccurate, based on the respective first and second confidence values.This process of assigning confidence values and generating a convergeddepth data point based on the confidence values and on other respectivedepth data points may be performed for depth data points correspondingto any and/or every particular physical point included in the pluralityof physical points on the surfaces of the objects. In this way, aconverged depth map representative of the surfaces of the objects may beformed that may be more accurate and/or otherwise superior to both oreither of the first and second depth maps. Examples of generating depthdata points based on other depth data points and respective confidencevalues to form converged depth maps will be provided below.

As mentioned above, it will be understood that additional facilities notexplicitly shown in FIG. 1 may also be included within system 100 as mayserve a particular implementation. For example, in certain embodiments,system 100 may include or may be associated with one or more facilitiesconfigured to generate, process, distribute, transmit, store, load, orotherwise manage or handle depth data representative of the surfaces ofthe objects included in the real-world scene. Such facilities withinsystem 100, or systems associated with system 100 that include suchfacilities, may distribute depth data accessed or generated byfacilities 102 or 104 and/or perform additional processing on the depthdata to convert the depth data into a useful form such as acomprehensive converged depth map of part or all of the real-worldscene, a dynamic volumetric model of the surfaces of the objectsincluded in the real-world scene, renderable virtual reality contentthat mimics the real-world scene, or the like. Specifically, forexample, a data stream representative of a dynamic volumetric model ofthe surfaces of the objects included in the real-world scene may begenerated. The dynamic volumetric model of the surfaces of the objectsin the real-world scene may be configured to be used to generate virtualreality media content representative of the real-world scene asexperienced from a dynamically selectable viewpoint corresponding to anarbitrary location within the real-world scene. For example, thedynamically selectable viewpoint may be selected by a user of a mediaplayer device while the user is experiencing the real-world scene usingthe media player device. These types of additional facilities orassociated systems may also provide, to the media player device based onthe generated data stream, the virtual reality media contentrepresentative of the real-world scene as experienced from thedynamically selectable viewpoint corresponding to the arbitrary locationwithin the real-world scene.

Storage facility 106 may maintain depth data 108 and/or any other datareceived, generated, accessed, managed, maintained, used, and/ortransmitted by facilities 102, 104, or any other facilities that may beincluded in system 100 in a particular implementation. Depth data 108may include depth data representative of the surfaces of the objectsincluded in the real-world scene (e.g., accessed or captured by depthmap access facility 102, generated by depth map convergence facility104, etc.). Examples of depth data will be provided and illustratedbelow. In some examples, along with depth data 108, storage facility 106may further include other data, such as data representative of avolumetric model (e.g., a real-time, 4D model) of the real-world scene,any part of which may be presented to a user from any arbitraryviewpoint selected by the user. As such, system 100 may provide virtualreality media content representative of the real-world event asexperienced from a dynamically selectable viewpoint corresponding to anarbitrary location at the real-world event by providing different partsof depth data 108 and/or other data included within storage facility 106to different media player devices based on dynamically selectableviewpoints that are selected by different respective users of the mediaplayer devices. Storage facility 106 may further include any other dataas may be used by facilities 102, 104, or other additional facilities togenerate depth data by converging independently-captured depth mapsand/or to create or distribute a volumetric representation of thereal-world scene in any way as may serve a particular implementation.

In order to generate depth data by converging independently-captureddepth maps as described herein, system 100 may use (e.g., include,communicate with, etc.) one or more depth map capture subsystemsconfigured to analyze a real-world scene to capture depth datarepresentative of surfaces of objects included within the real-worldscene.

To illustrate, FIG. 2A shows an exemplary configuration 200 in whichdepth data representative of exemplary surfaces of an object 202 in areal-world scene is captured by an exemplary depth map capture subsystem204. Configuration 200 illustrates a top view of depth map capturesubsystem 204 along with object 202.

Object 202 may be included within a real-world scene (not explicitlydemarcated in FIG. 2A) and may represent any type of object describedherein. For example, while object 202 is drawn as a relatively simplegeometric shape for the sake of clarity, it will be understood thatobject 202 may represent various types of objects having various levelsof complexity. Rather than a geometric shape, for instance, object 202could represent a person or another living thing, a non-transparentsolid, liquid, or gas, a less discrete object such as a wall, a ceiling,a floor, or any other type of object described herein or as may serve aparticular implementation.

As shown, object 202 may include various surfaces that may each reflectlight such that depth map capture subsystem 204 may capture depth datarepresentative of the surfaces by, for example, detecting, estimating,or otherwise determining the depth data in accordance with a particulardepth map capture technique such as those described herein. While object202 is relatively simple, the depth of the surfaces of object 202 mayappear different based on a position from which the depth of thesurfaces is captured (e.g., a fixed position of depth map capturesubsystem 204 with respect to object 202). In other words, object 202may look different based on a perspective or position from which object202 is viewed. Accordingly, to fully model object 202, depth datarepresentative of object 202 from various perspectives relative to(e.g., surrounding) object 202 may be used.

Depth map capture subsystem 204 may include any suitable hardware orcombination of hardware and software configured to capture a depth mapof object 202 from a fixed position at which depth map capture subsystem204 is disposed. More specifically, depth map capture subsystem 204 mayinclude hardware devices such as optical emitters (e.g., lasers or otherdevices for generating stimulated emission of electromagnetic radiationat a suitable frequency, camera flash equipment or other devices forgenerating pulses of light to bathe a real-world scene in light, etc.),optical sensors (e.g., video cameras, infrared (“IR”) sensors,time-of-flight sensors, etc.), and other hardware equipment configuredto perform at least one depth map capture technique for capturing adepth map representative of surfaces of objects (e.g., such as object202) within a real-world scene. Depth map capture subsystem 204 mayfurther include software associated with any of the devices or equipmentmentioned above or that is configured to run on a general or specificpurpose processor included within depth map capture subsystem 204. Assuch, depth map capture subsystem 204 may be configured to perform oneor more depth map capture techniques to capture depth datarepresentative of object 202. Specific examples of depth map capturetechniques that may be performed by depth map capture subsystem 204 willbe described below.

In certain examples, depth map capture subsystem 204 may be includedwithin system 100 such that accessing a depth map by system 100 includescapturing the depth map using depth map capture subsystem 204. In otherexamples, depth map capture subsystem 204 may be separate from system100 (e.g., included within a separate system communicatively coupledwith or otherwise associated with system 100) such that accessing thedepth map by system 100 includes receiving the depth map from depth mapcapture subsystem 204 (e.g., after depth map capture subsystem 204 hascaptured and transmitted the depth map to system 100). As such, depthmap capture subsystem 204 may perform additional functionality describedabove with respect to system 100.

For example, along with capturing or otherwise facilitating theaccessing of at least one depth map, depth map capture subsystem 204 mayalso assign or facilitate assigning confidence values to depth datapoints included within the depth map. Specifically, for instance, depthmap capture subsystem 204 may analyze data captured from the real-worldscene to identify one or more attributes of various surfaces of object202 in order to determine a confidence (e.g., a relative likelihood, adegree of certainty, etc.) that depth data captured to representparticular physical points on the surfaces of object 202 is accurate.Similarly, depth map capture subsystem 204 may analyze a viewingperspective associated with the fixed position at which depth mapcapture subsystem 204 is disposed to further assess the confidence thatthe depth data captured to represent the particular physical points isaccurate. Examples of determining and assigning confidence values willbe described below.

Depth map capture subsystem 204 may be associated with a particularscope of capture 206, illustrated in configuration 200 by dotted linesemanating from depth map capture subsystem 204 to broadly encompassobject 202 and an area around object 202. Scope of capture 206 may belimited by various characteristics of depth map capture subsystem 204and/or the equipment and devices included within depth map capturesubsystem 204. For example, if depth map capture subsystem 204 includesone or more optical sensors that have a finite angle of view (e.g., a90° by 60° angle of view), the finite angle of view of the opticalsensors may limit scope of capture 206 of depth map capture subsystem204 such that scope of capture 206 includes parts of a real-world scenein front of depth map capture subsystem 204, but not necessarily all theway around (e.g., to the sides and/or behind) depth map capturesubsystem 204. Scope of capture 206 may be any scope as may serve aparticular implementation. For example, while configuration 200illustrates a limited, directional scope of capture 206 in front ofdepth map capture subsystem 204, other examples may include a 360° scopeof capture (e.g., an annular ring around depth map capture subsystem 204or a complete sphere around depth map capture subsystem 204) or anyother suitable scope of capture as may serve a particularimplementation.

FIG. 2B illustrates a perspective view 208 of object 202 from theperspective of depth map capture subsystem 204. In view 208, varioussurfaces of object 202 (i.e., surfaces 210-1 through 210-5) are shown.It will be understood that other surfaces of object 202 may also existthat may not be visible from the perspective of view 208 (i.e., from avantage point of a fixed position at which depth map capture subsystem204 is disposed). Each surface 210 may include a plurality of physicalpoints for which depth data may be captured by depth map capturesubsystem 204. For example, an exemplary physical point 212 is calledout on surface 210-4. Physical point 212 will be referred to below tohelp explain how system 100 may generate depth data by independentlycapturing and then converging depth maps. However, it will be understoodthat physical point 212 is an arbitrary, exemplary point only. Theprinciples described with respect to physical point 212 may apply to anyand all physical points on any and all surfaces 210 of object 202, aswell as to other physical points on other surfaces of other objectsincluded within the real-world scene.

The surfaces of one or more objects that may be captured by a depth mapcapture subsystem of a depth data generation system may be determinedbased on a fixed position with respect to the objects at which the depthmap capture subsystem is disposed. For example, as illustrated in FIG.2B, depth map capture subsystem 204 may capture depth data pointrepresentative of particular physical points (e.g., such as physicalpoint 212) on surfaces 210-1 through 210-5 of object 202 because thefixed position and orientation of depth map capture subsystem 204 issuch that scope of capture 206 faces surfaces 210-1 through 210-5. Inorder to capture depth data representative of all the surfaces of object202, as well as to capture depth data representative of other objectswithin the real-world scene and/or to capture the depth data fromperspectives associated with relatively high confidence values (e.g.,perspectives from which surfaces can be analyzed relatively“straight-on” rather than at a sharp angle), a plurality of depth mapcapture subsystems similar to depth map capture subsystem 204 may beused. For example, the plurality of depth map capture subsystems may bedistributed across a plurality of nodes included within a configurationof system 100, each of the nodes disposed at different fixed nodepositions (e.g. different fixed node positions varying along any axis of3D space or varying in distance from a real-world scene) in a pluralityof fixed node positions with respect to (e.g., surrounding or partiallysurrounding) the real-world scene.

To illustrate, FIG. 3 shows an exemplary implementation 300 of system100 positioned with respect to an exemplary real-world scene in order togenerate depth data by converging independently-captured depth maps.More specifically, implementation 300 of system 100 includes a pluralityof nodes 302 (i.e., nodes 302-1 through 302-8) disposed at fixed nodepositions with respect to (e.g., in this case, surrounding) a real-worldscene 304 that includes object 202, described above in relation to FIGS.2A and 2B.

Each of nodes 302 may include or implement one or more depth map capturesubsystems such as depth map capture subsystem 204 described above withrespect to system 200. For example, each node 302 may include equipmentand/or devices for performing at least one depth map capture technique(described in more detail below) to capture a depth map, hardware and/orsoftware for determining and assigning respective confidence values toeach depth data point included within the depth map, and/or otherelements described above with respect to depth map capture subsystem 204as may serve a particular implementation. Additionally, sinceimplementation 300 is an implementation of system 100, one or moreelements described above with respect to system 100 or described belowwith respect to other implementations of system 100 may also be includedwithin one or more of nodes 302.

Accordingly, each node 302 may be configured to perform operations suchas those described and illustrated above with respect to system 100and/or system 200. Specifically, for example, each node 302 may access(e.g., capture, receive, load, etc.) first and second depth mapsrepresentative of different physical points included in a plurality ofphysical points on the surfaces of objects such as object 202 includedin real-world scene 304. Additionally, each node 302 may be configuredto converge two or more depth maps (e.g., the first and second depthmaps) into a converged depth map representative of the physical pointsincluded in the plurality of physical points on the surfaces of objectsin real-world scene 304.

As shown, because of the different fixed node positions of nodes 302 ofimplementation 300, each node 302 may be associated with a uniqueperspective of object 202 such that the surfaces of object 202 may bedetected from various perspectives surrounding object 202 and each node302 may detect characteristics of the surfaces of object 202 that wouldbe difficult or impossible to detect from the fixed node positions ofother nodes 302. To illustrate, each node 302 includes dotted linesemanating therefrom representative of the scope of capture of theparticular node 302. Specifically, for example, a scope of capture 306of node 302-1 is explicitly labeled in FIG. 3. In the setup ofimplementation 300, each of nodes 302 may be positioned so as to captureall or substantially all of the circular area designated as real-worldscene 304 from the perspective (i.e., angle, distance, etc.) afforded bythe respective fixed node position of the node. For example, all of therespective areas of nodes 302 may be overlapping with the respectiveareas of all the other nodes 302 in an area (e.g., a circular area)designated as real-world scene 304.

It will be understood, however, that in other examples, a real-worldscene may not be circular and each of nodes 302 may not capture all orsubstantially all of the real-world scene from a particular perspective.For example, a real-world scene may be round (e.g., circular,elliptical, etc.) or non-round (e.g., a shape having corners such as atriangle, square, or other polygon). Additionally, as will beillustrated below with respect to FIG. 5, a real-world scene may beelongated such that one side of the real-world scene may besignificantly longer than another (e.g., rectangular like a basketballcourt, stretched out like a racetrack, etc.). Accordingly, in certainexamples, each node of an implementation of system 100 may be associatedwith an area that includes a portion (e.g., a horizontal portion, avertical portion, etc.) of the real-world scene that is smaller than theentire real-world scene. As such, various portions of the real-worldscene associated with each node may overlap with other portions of thereal-world scene (e.g., portions of the real-world scene associated withneighboring nodes) but may not necessarily overlap with every otherportion of the real-world scene associated with every other node.

In FIG. 3, a plurality of eight nodes 302 are illustrated to besurrounding real-world scene 304 and object 202. It will be understoodthat this number of nodes is exemplary only and that any number of nodes302 as may serve a particular implementation may be used in variousexamples. Additionally, while nodes 302 are illustrated as completelysurrounding real-world scene 304, it will be understood that, in certainimplementations, nodes 302 may be located in fixed node positions withrespect to real-world scene 304 that do not necessarily surroundreal-world scene 304. For example, if real-world scene 304 represents astage where a play is being performed, nodes 302 may be located in fixednode positions with respect to real-world scene 304 in front of thestage but may not completely surround the stage in back. In certainexamples, real-world scene 304 may include several areas (e.g.,geographical areas) of particular interest to users along with otherareas of relatively less interest. As such, nodes 302 may be distributedto cover several distinct (i.e., non-touching) areas. For example,real-world scene 304 may include a racetrack that is several miles longand nodes 302 may be disposed at fixed node positions associated onlywith particular turns of the racetrack and/or the starting line, thefinish line, the pits, and/or other areas of interest along theracetrack.

Nodes 302 may be communicatively coupled by a connection 308 (e.g.,which may represent any wired or wireless direct or network connectionas may serve a particular implementation) to one another and/or toanother device such as to a data capture processing unit 310. This mayallow nodes 302 to maintain synchronicity in time, position, angle, etc.so that a dynamic volumetric model of the surfaces of objects includedwithin real-world scene 304 (e.g., including object 202) may begenerated. For example, nodes 302 may send and receive timing signals toensure that each node 302 detects corresponding data at the same timeand that the data detected by different nodes 302 may be timestampedwith a universal time shared by all of nodes 302 in system 100. In otherembodiments, audio, video, and/or other cues may be used by each node302 to ensure that each node 302 detects corresponding data at the sametime.

Data capture processing unit 310 may either be included within orcommunicatively coupled to implementation 300 of system 100 as may servea particular implementation. Data capture processing unit 310 mayinclude one or more computing resources configured to assign confidencevalues to particular depth data points included within depth maps, togenerate additional depth data points based on the particular depth datapoints and/or the confidence values assigned to the particular depthdata points, and/or to otherwise converge independently-captured depthmaps into converged depth maps as may serve a particular implementation.For example, data capture processing unit 310 may include one or moreservers, desktop computers, or other computing devices that may leveragevarious types of hardware (e.g., central processing units (“CPUs”),field programmable gate arrays (“FPGAs”), general purpose graphicsprocessing units (“GPGPUs”), etc.) and/or software to perform one ormore of the operations described herein. In some examples, data captureprocessing unit 310 may be configured to perform parallel computingoperations. For instance, data capture processing unit 310 may performparallel computing operations by simultaneously employing multiple typesof hardware (e.g., FPGAs and GPGPUs) to perform hardware-acceleratedparallel computing, by coordinating multiple hardware instances (e.g.,multiple GPGPUs on different desktop computers, etc.) to performmultiple-hardware parallel computing, by using a message passinginterface (“MPI”) to coordinate multiple computing nodes (e.g., eachcontaining a plurality of GPGPUs or other hardware instances) to performmultiple-node parallel computing, and/or by any other method as mayserve a particular implementation.

While data capture processing unit 310 may be configured to generatedepth data by converging independently-captured depth maps intoconverged depth maps, it is also noted (as mentioned above) that depthmap capture subsystems (e.g., included within nodes 302) may similarlyinclude computing resources configured to generate depth data byconverging independently-captured depth maps.

As such, in certain examples, the depth data represented within theconverged depth maps may be generated in an integrated manner by acomputing system such as data capture processing unit 310, while, inother examples, the depth data may be generated in a distributed mannerby each node 302 before being transmitted to data capture processingunit 310. In yet other examples, depth data may be converged in ahierarchy of levels and/or by a plurality of computing devices includingcomputing devices associated with each node 302 as well as computingdevices associated with data capture processing unit 310. Specifically,system 100 may access (e.g., using data capture processing unit 310) afirst depth map by converging (e.g., using one or more of nodes 302) afirst plurality of antecedent depth maps into the first depth map, wherethe antecedent depth maps in the first plurality of antecedent depthmaps are each captured by way of a first depth map capture technique atthe one or more nodes 302. Similarly, system 100 may access (e.g., usingdata capture processing unit 310) a second depth map by converging(e.g., using the one or more nodes 302) a second plurality of antecedentdepth maps into the second depth map, where the antecedent depth maps inthe second plurality of antecedent depth maps are each captured by wayof a second depth map capture technique (e.g., the same or a differentdepth map capture technique as the first depth map capture technique) atthe one or more nodes 302.

As a more specific example, node 302-1 may be configured to generatedepth data by converging independently-captured depth maps captured bydepth map capture subsystems included within node 302-1 into a firstconverged depth map. Similarly, node 302-2 may be configured to generatedepth data by converging independently-captured depth maps captured bydepth map capture subsystems included within node 302-2 into a secondconverged depth map. Nodes 302-1 and 302-2 may transmit the respectivegenerated depth data (i.e., the first and second converged depth maps,respectively) to data capture processing unit 310 (e.g., by way ofconnection 308). Data capture processing unit 310 may then access thefirst and second converged depth maps (e.g., by receiving the convergeddepth maps from nodes 302-1 and 302-2) and treat the first and secondconverged depth maps as antecedent depth maps for an additionalconverged depth map. Specifically, data capture processing unit 310 maygenerate additional depth data by converging the first and secondconverged depth maps (or at least the portions of them that overlap)into a third converged depth map.

By converging various parts of various depth maps from various nodes 302in this way, system 100 may ultimately generate a depth maprepresentative of many or all of the objects within real-world scene 304from various angles surrounding the objects so that many or all of thesurfaces of the objects are represented in a unified converged depthmap, even if depth data for all of the objects and/or all of thesurfaces would be impossible to capture from a single fixed nodeposition.

After generating and/or otherwise processing the depth datarepresentative of the surfaces of object 202 included in real-worldscene 304, data capture processing unit 310 may use the depth data orprovide the depth data for use by another system included within orotherwise associated with system 100 in any way as may serve aparticular implementation. For example, based on the generated depthdata (e.g., one or more converged depth maps, a unified converged depthmap representative of the entirety of real-world scene 304, etc.),system 100 may generate a dynamic volumetric model of the surfaces ofobject 202 within real-world scene 304. A dynamic volumetric model of anobject may include and/or be generated based both on 1) the depth datarepresenting where and how the object is positioned in 3D space at aparticular time, or with respect to time over a particular time period,and on 2) synchronous 2D video data (e.g., captured by system 100 oranother system associated with system 100) mapped onto a positionalmodel (e.g., a wireframe model of the object derived from the depthdata) to represent how the object appeared at the particular time orwith respect to time over the particular time period. As such, dynamicvolumetric models may be 3D models including three spatial dimensions orfour-dimensional (“4D”) models that include the three spatial dimensionsas well as a temporal dimension.

In some examples, system 100 may further generate a data stream (e.g., areal-time data stream) representative of the dynamic volumetric model ofthe surfaces of object 202 included in real-world scene 304. Forexample, data capture processing unit 310 may generate the data streamin real time such that users not physically located within or aroundreal-world scene 304 may be able to experience real-world scene 304live, in real time or near-real time, via virtual reality media contentrepresentative of real-world scene 304. Accordingly, the dynamicvolumetric model of the surfaces of object 202 may be configured to beused to generate virtual reality media content representative ofreal-world scene 304. The virtual reality media content may be generatedby system 100 (e.g., by data capture processing unit 310) and/or byanother system associated with system 100 (e.g., another system operatedby a virtual reality media provider or by a separate entity such as avirtual reality media content distributor associated with the virtualreality media provider). Virtual reality media content may be generated(e.g., based on a real-time data stream generated from a dynamicvolumetric model of the surfaces of object 202 and/or other objectswithin real-world scene 304) by implementation 300 (e.g., nodes 302,data capture processing unit 310, etc.), and then distributed by way ofa network 312 to one or more media player devices such as a media playerdevice 314 associated with a user 316. It will be understood that incertain implementations network 312 and/or media player device 314 maybe communicatively coupled to system 100 (e.g., to data captureprocessing unit 310), but may be separate from (i.e., not incorporatedor included within) system 100.

System 100 may provide the virtual reality media content to media playerdevice 314 so that user 316, who may not be physically located nearreal-world scene 304 but who may wish to experience real-world scene 304(e.g., a real-world event occurring within real-world scene 304), mayexperience real-world scene 304 virtually using media player device 314.Additionally, it may be desirable for user 316 to experience real-worldscene 304 live (e.g., in real time or near-real time as a real-worldevent is occurring). Accordingly, system 100 may provide the virtualreality media content representative of real-world scene 304 to mediaplayer device 314 in real time.

While data processing and data distribution may take a finite amount oftime such that it may be impossible for a user to experience real-worldscene 304 precisely as events within real-world scene 304 occur, as usedherein, an operation (e.g., providing the virtual reality media content)is considered to be performed “in real time” when the operation isperformed immediately and without undue delay. Accordingly, a user maybe said to experience a real-world scene in real time even if the userexperiences particular occurrences within the event (e.g., a particularshot in a basketball game) a few seconds or minutes after theoccurrences actually take place. To support real-time dynamic volumetricmodeling and experiencing of immersive virtual reality worlds based onlive real-world scenes, system 100 or certain components of system 100(e.g., data capture processing unit 310) may include or be implementedby powerful hardware resources (e.g., multiple servers includingmultiple processing units) that may be configured to perform the immenseprocessing required for real-time creation and distribution of immersivevirtual reality worlds based on real-time data streams representative ofdynamic volumetric models of the surfaces of objects within real-worldscenes.

It may be undesirable for user 316, who may experience real-world scene304 virtually (e.g., using media player device 314 to present virtualreality media content provided by system 100), to be limited to one ormore discrete positions within the immersive virtual reality worldrepresentative of real-world scene 304. As such, system 100 may providethe virtual reality media content representative of real-world scene 304as experienced from a dynamically selectable viewpoint corresponding toan arbitrary location within real-world scene 304. The dynamicallyselectable viewpoint may be selected by user 316 while user 316 isexperiencing real-world scene 304 using media player device 314.

As used herein, an “arbitrary location” may refer to any point in spaceat the real-world event. For example, arbitrary locations are notlimited to fixed node positions (e.g., where nodes 302 are disposed)around real-world scene 304, but also include all the positions betweennodes 302 and even positions where nodes such as nodes 302 may not beable to be positioned (e.g., in the middle of real-world scene 304).Moreover, arbitrary locations may not be limited to aligning with aviewing angle of any particular node 302. In some examples, sucharbitrary locations (i.e., that do not directly align with a viewingangle of any node 302) may correspond to the most desirable viewpointswithin real-world scene 304. For instance, if real-world scene 304includes a basketball game, nodes 302 may not be allowed to bepositioned in the middle of the basketball court because nodes 302 wouldinterfere with gameplay of the basketball game. However, user 316 maydynamically select viewpoints from which to experience the game that arein any arbitrary location on the basketball court. For example, the usermay dynamically select his or her viewpoint to follow the basketball upand down the basketball court and experience the basketball game as ifstanding on the basketball court in the middle of the action of thegame. In other words, for example, while nodes 302 may be positioned atfixed node positions surrounding the basketball court, but may not bepositioned directly on the court so as not to interfere with gameplay ofthe basketball game, user 316 may dynamically select viewpoints fromwhich to experience the game that are in any arbitrary location on thebasketball court.

Network 312 may include any provider-specific wired or wireless network(e.g., a cable or satellite carrier network or a mobile telephonenetwork), the Internet, wide area network, or any other suitablenetwork. Data may flow between system 100 (e.g., by way of data captureprocessing unit 310) and media player device 314 (as well as other mediaplayer devices not explicitly shown) using any communicationtechnologies, devices, media, and protocols as may serve a particularimplementation. For example, data capture processing unit 310 maycommunicate with media player device 314 using any suitablecommunication technologies, devices, media, and/or protocols supportiveof data communications, including, but not limited to, socketconnections, Ethernet, data bus technologies, data transmission media,communication devices, Transmission Control Protocol (“TCP”), InternetProtocol (“IP”), File Transfer Protocol (“FTP”), Telnet, HypertextTransfer Protocol (“HTTP”), HTTPS, Session Initiation Protocol (“SIP”),Simple Object Access Protocol (“SOAP”), Extensible Mark-up Language(“XML”) and variations thereof, Real-Time Transport Protocol (“RTP”),User Datagram Protocol (“UDP”), Global System for Mobile Communications(“GSM”) technologies, Code Division Multiple Access (“CDMA”)technologies, Evolution Data Optimized Protocol (“EVDO”), 4G Long TermEvolution (“LTE”), Voice over IP (“VoIP”), Voice over LTE (“VoLTE”),WiMax, Time Division Multiple Access (“TDMA”) technologies, ShortMessage Service (“SMS”), Multimedia Message Service (“MMS”), radiofrequency (“RF”) signaling technologies, wireless communicationtechnologies (e.g., Bluetooth, Wi-Fi, etc.), in-band and out-of-bandsignaling technologies, and other suitable communications technologies.While only one network 312 is shown to interconnect data captureprocessing unit 310 and media player device 314 in FIG. 3, it will berecognized that data capture processing unit 310, media player device314, and/or other subsystems of system 100 or systems associated withsystem 100 may intercommunicate by way of multiple interconnectednetworks as may serve a particular implementation.

Media player device 314 may be used by user 316 to access and experiencevirtual reality media content received from system 100 (e.g., from datacapture processing unit 310). To this end, media player device 314 mayinclude or be implemented by any device capable of presenting a field ofview of an immersive virtual reality world (e.g., an immersive virtualreality world representative of real-world scene 304) and detecting userinput from user 316 to dynamically update the immersive virtual realityworld presented within the field of view as user 316 experiences theimmersive virtual reality world. Exemplary immersive virtual realityworlds and fields of view will be illustrated and described in moredetail below. Additionally, in certain implementations, media playerdevice 314 may further be configured to present sensory data (e.g., 3Dsensory data other than video and audio data presented in connectionwith the immersive virtual reality world) along with the field of viewof the immersive virtual reality world. For example, media player device314 may include, connect to, or otherwise be associated with sensoryfeedback devices such as sensory feedback gloves, sensory feedback bodysuits, and the like, which may present the sensory data to provide userswith a sensation of feeling, touching, smelling, or otherwise perceivingparticular objects or other elements of the immersive virtual realityworld and thereby enhance users' experiences within the immersivevirtual reality world. As such, in certain examples, system 100 (e.g.,data capture processing unit 310 and/or other components of system 100)may receive, generate, process, transmit, and/or otherwise providesensory data to media player device 314 to allow or facilitate enhancedsensory experiences for users within the immersive virtual realityworld.

In some examples, media player device 314 may be configured to allowuser 316 to select respective virtual reality media content programs(e.g., associated with different real-world scenes, as well as othertypes of virtual reality media content programs) that user 316 may wishto experience. In certain examples, media player device 314 may downloadvirtual reality media content programs that user 316 may experienceoffline (e.g., without an active connection to data capture processingunit 310). In other examples, media player device 314 may request andreceive data streams representative of virtual reality media contentprograms that user 316 experiences while media player device 314 remainsin active communication with system 100 (e.g., data capture processingunit 310) by way of network 312.

Media player device 314 may take one of several different form factors.For example, media player device 314 may include or be implemented by ahead-mounted virtual reality device (e.g., a virtual reality gamingdevice) that includes a head-mounted display screen, by a personalcomputer device (e.g., a desktop computer, laptop computer, etc.), by amobile or wireless device (e.g., a smartphone, a tablet device, a mobilereader, etc.), or by any other device or configuration of devices thatmay serve a particular implementation to facilitate receiving and/orpresenting virtual reality media content. Different types of mediaplayer devices (e.g., head-mounted virtual reality devices, personalcomputer devices, mobile devices, etc.) may provide different types ofvirtual reality experiences having different levels of immersion foruser 316.

While, as mentioned above, it may be desirable in some examples for user316 to experience real-world scene 304 in real time (e.g., as eventsoccur within real-world scene 304 or after a trivial period of delay),in other examples, user 316 may wish to experience real-world scene 304in a time-shifted manner, rather than in real time. To this end, system100 may store and maintain, subsequent to providing virtual realitymedia content representative of real-world scene 304 in real time, arecording of the data stream representative of the dynamic volumetricmodel of the surfaces of the objects of real-world scene 304. Then, whenuser 316 later wishes to experience real-world scene 304, system 100 mayprovide virtual reality media content representative of real-world scene304 as experienced from a dynamically selectable viewpoint correspondingto an arbitrary location within real-world scene 304 selected by theuser. For example, system 100 may provide the virtual reality mediacontent to media player device 314 based on the recording of thereal-time data stream.

To illustrate a node in which one or more depth map capture subsystemsare used to independently capture one or more depth maps and/or togenerate depth data by converging the independently-captured depth maps(i.e., a node similar or the same as one of nodes 302), FIG. 4 shows anexemplary node 400 of an exemplary implementation of system 100. Node400 may include a plurality of camera rigs that each include a pluralityof depth map capture subsystems. Specifically, as shown in FIG. 4, incertain implementations, node 400 may include three camera rigs 402(i.e., camera rigs 402-1, 402-2, and 402-3), which may each include arespective plurality of depth map capture subsystems. Each of the depthmap capture subsystems may include various hardware devices and/or othersuitable equipment for performing a depth map capture technique and/orany other functionality as may be described herein or as may serve aparticular implementation. As shown, camera rigs 402 may be supportedand or positioned (e.g., in respective fixed positions) by nodepositioning structure 404, such as a tripod or the like.

Node positioning structure 404 may provide flexibility in how camerasand other devices are positioned by allowing camera rigs 402 to beadjusted to various heights, angles, etc., based on characteristics of aparticular real-world scene being captured (e.g., characteristics of thetypes of objects within the real-world scene, etc.). For example, if areal-world scene includes human subjects in standing or sittingpositions (e.g., such as a basketball game), node positioning structure404 may allow one camera rig 402 to be positioned at a height ofapproximately eight feet (i.e., taller than most people) and angled tobe aiming slightly downward to capture data related to the tops of theheads and shoulders of the human subjects. Similarly, node positioningstructure 404 may also allow another camera rig 402 to be positioned ata height of approximately two feet and angled to be aiming slightlyupward to capture data related to the bottoms of the chins of the humansubjects, and so forth. These heights and angles are exemplary only. Itwill be understood that node positioning structure 404 may supportcamera rigs 402 being positioned in any suitable arrangement as mayserve a particular implementation.

Node 400 may represent any node of system 100 as may serve a particularimplementation. For example, any of nodes 302 illustrated in FIG. 3 maybe set up with multiple camera rigs to resemble node 400 in certainimplementations. Additionally, any or all of the other nodes describedin various implementations described below may similarly resemble node400 or a variant of node 400. For example, while node 400 is shown toinclude three camera rigs 402 each with two depth map capturesubsystems, variants of node 400 may have any number of camera rigs eachincluding any number or configuration of depth map capture subsystems orcomponents thereof as may serve a particular implementation.

While node 400 may be located at a fixed node position with respect tothe real-world scene such that a horizontal perspective of each camerarig 402 is aligned, camera rigs 402 are spaced apart along a verticaldimension such that each depth map capture subsystem (or set of depthmap capture subsystems, as the case may be) may have a slightlydifferent perspective (e.g., based on the distinct fixed positions ofthe components within each depth map capture subsystem) than other depthmap capture subsystems (or sets of depth map capture subsystems). Thus,for example, a depth map capture subsystem of camera rig 402-1 may moreaccurately and/or effectively detect surfaces of objects that are higheroff the ground (e.g., the tops of people's heads, etc.) than, forexample, a depth map capture subsystem of camera rig 402-3, while thedepth map capture subsystem of camera rig 402-3 may excel in accuratelyand effectively detecting surfaces of objects nearer to the ground.

As another example of how system 100 may operate and/or be used in avirtual reality application in order to generate and/or facilitategeneration of a dynamic volumetric model of surfaces of objects in areal-world scene, FIG. 5 illustrates an exemplary implementation 500 ofsystem 100 positioned with respect to an exemplary real-world scene 502in order to generate depth data by converging independently-captureddepth maps. As shown, real-world scene 502 in FIG. 5 includes or isassociated with a real-world event of a basketball game (e.g., a livebasketball game).

Similar to real-world scene 304 in FIG. 3, real-world scene 502 of FIG.5 is shown to be surrounded by inward-facing synchronous nodes 504-i.Additionally, real-world scene 502 is shown to surround at least oneoutward-facing node 504-o. Nodes 504-i and 504-o may collectively bereferred to herein as “nodes 504”). The fixed node positions at whichnodes 504 are located illustrate an example where each node 504 may beassociated with only a particular slice (i.e., a horizontal slice) of areal-world scene, rather than the entirety of the real-world scene, aswas described above. Specifically, each node 504 may capture (e.g.,along with neighboring nodes 504 and/or nodes 504 that are across thebasketball court) only a particular section of real-world scene 502based on the location and orientation of the fixed node position atwhich the node is disposed.

Nodes 504 may be configured to capture one or more depth maps (e.g.,independently-captured depth maps) representative of surfaces of objects506 included in real-world scene 502 (e.g., players, a basketball 508,etc.) by way of, for example, one or more depth map capture techniques.Accordingly, nodes 504 may be similar to nodes 302 in that each node 504may include one or more depth map capture subsystems and/or additionalcomponents described to be associated with other nodes described herein(e.g., node 400) or as may serve a particular implementation.

Additionally, nodes 504 may also include features not explicitlydescribed in relation to nodes 302 above. For example, nodes 504 mayinclude video capture devices (e.g., visible light video cameras, etc.)configured to capture texture data (e.g., 2D video data) of objects 506included in real-world scene 502 that, when combined with depth datarepresentative of objects 506, may be used to generate dynamicvolumetric models of the surfaces of objects 506 included in real-worldscene 502. Also, as illustrated by node 504-0, one or more of nodes 504may be outward-facing nodes that capture depth maps for one or moreobjects surrounding real-world scene 502. For example, node 504-0 mayfacilitate capturing depth data and/or texture data representative ofobjects in the spectator seating areas at the venue in which thebasketball game is taking place. Because node 504-o may not be able tobe positioned directly on the basketball court (i.e., because it wouldinterfere with the basketball game), node 504-o may be suspended abovereal-world scene 502 or otherwise positioned as may serve a particularimplementation.

Objects 506 may include any objects associated with (e.g., located in oraround) real-world scene 502. For example, objects 506 may includepeople on the court (e.g., basketball players, referees, and otherpeople on the basketball court), basketball 508, and/or other livingand/or inanimate objects such as basketball standards (i.e., backboards,rims, nets, etc.), the floor of the basketball court, people and/orobjects on the sidelines of the basketball game, spectators and seatingareas surrounding the basketball court, and the like.

As shown, nodes 504 may be communicatively coupled by connections 510(e.g., including wired or wireless connections such as described abovein relation to connection 308) to one another and/or to data captureprocessing unit 310, which was described above in relation to FIG. 3. Asalso described above, data capture processing unit 310 may becommunicatively coupled by network 312 to media player device 314, whichis associated with user 316. Based on depth data generated by datacapture processing unit 310, system 100 (e.g., data capture processingunit 310 or another component of system 100) may generate a data stream(e.g., a real-time data stream) representative of a dynamic volumetricmodel of the surfaces of objects 506 included in real-world scene 502.Data capture processing unit 310 may further generate virtual realitymedia content representative of real-world scene 502 (e.g., based on thedata stream) and provide the virtual reality media content to mediaplayer device 314 to be experienced by user 316, as described above.

To illustrate, FIG. 6 shows an exemplary virtual reality experience 600in which user 316 is presented with exemplary virtual reality mediacontent representative of real-world scene 502 as experienced from adynamically selectable viewpoint corresponding to an exemplary arbitrarylocation within real-world scene 502. Specifically, virtual realitymedia content 602 is presented within a field of view 604 that showsreal-world scene 502 from a viewpoint corresponding to an arbitrarylocation right underneath a basketball standard within real-world scene502 where a shot is being made. An immersive virtual reality world 606based on the real-world event may be available for the viewer toexperience by providing user input (e.g., head movements, keyboardinput, etc.) to look around and/or to move around (i.e., dynamicallyselect a viewpoint from which to experience) immersive virtual realityworld 606.

For example, field of view 604 may provide a window through which user316 may easily and naturally look around immersive virtual reality world606. Field of view 604 may be presented by media player device 314(e.g., on a display screen of media player device 314) and may includevideo depicting objects surrounding the user within immersive virtualreality world 606. Additionally, field of view 604 may dynamicallychange in response to user input provided by the user as the userexperiences the immersive virtual reality world. For example, the mediaplayer device may detect user input (e.g., moving or turning the displayscreen upon which the field of view is presented). In response, thefield of view may display different objects and/or objects seen from adifferent viewpoint (e.g., a viewpoint corresponding to the position ofthe display screen) in place of the objects seen from the previousviewpoint.

In FIG. 6, immersive virtual reality world 606 is illustrated as asemi-sphere, indicating that user 316 may look in any direction withinimmersive virtual reality world 606 that is substantially forward,backward, left, right, and/or up from the viewpoint of the locationunder the basketball standard that user 316 has currently selected. Inother examples, immersive virtual reality world 606 may include anentire 360° by 180° sphere such that user 316 may also look down.Additionally, user 316 may move around to other locations withinimmersive virtual reality world 606 (i.e., dynamically selectingdifferent dynamically selectable viewpoints of the real-world event).For example, user 316 may select a viewpoint at half court, a viewpointfrom the free-throw line facing the basketball standard, a viewpointsuspended above the basketball standard, or the like.

As described above, system 100 may include one or more depth map capturesubsystems each configured to independently capture depth maps that maybe converged to form converged depth maps (e.g., converged depth mapsthat are more accurate or otherwise superior to the depth maps capturedby the depth map capture subsystems). As further mentioned above, eachdepth map capture subsystem may capture a respective depth map from aparticular fixed position with respect to a real-world scene at whichthe depth map capture subsystem is disposed, based on a depth mapcapture technique.

FIGS. 7A-7C show three exemplary depth map capture techniques that maybe implemented by one or more depth map capture subsystems within system100. Specifically, FIG. 7A illustrates a depth map capture technique700-A representing an exemplary structured light depth map capturetechnique, FIG. 7B illustrates a depth map capture technique 700-Brepresenting an exemplary stereoscopic depth map capture technique, andFIG. 7C illustrates a depth map capture technique 700-C representing anexemplary time-of-flight depth map capture technique. Collectively,depth map capture techniques 700-A, 700-B, 700-C, and other depth mapcapture techniques described herein or that may be used to serveparticular implementations are referred to herein as depth map capturetechniques 700.

In some examples, related depth map capture subsystems (e.g., depth mapcapture subsystems within the same camera rig, depth map capturesubsystems within the same node, depth map capture subsystems at thesame relative position on different camera rigs or different nodes,etc.) may perform depth map capture techniques that are complementary toone another. As used herein, “complementary” depth map capturetechniques may refer to depth map capture techniques that each have oneor more strengths that align, to at least some degree, with one or moreweaknesses of the other depth map capture technique. For example, aswill be described in more detail below, a first depth map capturetechnique 700 (e.g., depth map capture technique 700-A) may be adaptedto detect depth data representative of a first type of surface of anobject relatively effectively, while being relatively ineffective atdetecting depth data representative of a second type of surface.Conversely, a second depth map capture technique 700 (e.g., depth mapcapture technique 700-B) may be adapted to detect depth datarepresentative of the second type of surface relatively effectively,while being relatively ineffective at detecting depth datarepresentative of the first type of surface. Accordingly, the first andsecond depth map capture techniques may be complementary depth mapcapture techniques. It will be understood that each of depth map capturetechniques 700 may be complementary with every other depth map capturetechnique 700, as well as to other depth map capture techniques notexplicitly illustrated in FIGS. 7A-7C (e.g., such as an infraredstereo-disparity depth map capture technique described below).

In certain examples, system 100 may converge (e.g., using confidencevalues assigned and analyzed to ensure that the mosteffectively-captured depth data is relied on to the greatest extent, asdescribed in more detail below) depth maps captured using complementarydepth map capture techniques in order to take advantage of respectivestrengths of the complementary depth map capture techniques whileavoiding respective weaknesses of the depth map capture techniques. Inthe same or other examples, system 100 may also converge (e.g., alsousing the confidence values) depth maps captured from different fixedpositions with respect to the real-world scene (e.g., so as to havedifferent vantage points with which to detect depth data representativeof various surfaces of objects within the real-world scene) in order tosimilarly take advantage of respective strengths of each vantage pointwhile avoiding respective weaknesses of the vantage point.

Depth map capture techniques 700 may be performed by depth map capturesubsystems in any suitable way and/or using any configuration as mayserve a particular implementation. For example, referring to FIG. 4,each depth map capture subsystem on a particular camera rig 402 (e.g.,the two depth map capture subsystems on camera rig 402-1, for example)may perform a different depth map capture technique 700. Morespecifically, for instance, one depth map capture subsystem on camerarig 402-1 may perform structured light depth map capture technique 700-Awhile the other depth map capture subsystem on camera rig 402-1 mayperform stereoscopic depth map capture technique 700-B.

In some examples, each camera rig within a node (e.g., all three camerarigs 402 in node 400) may perform the same depth map capture techniques.In other examples, different camera rigs within a node may performdifferent depth map capture techniques. For example, referring to theexample above where the depth map capture subsystems of camera rig 402-1perform depth map capture techniques 700-A and 700-B, the depth mapcapture subsystems of another camera rig within node 400 (e.g., camerarig 402-2) may perform the same depth map capture techniques (e.g.,depth map capture techniques 700-A and 700-B) or different depth mapcapture techniques (e.g., depth map capture techniques 700-A and 700-C,700-B and 700-C, etc.). Similarly, just as various configurations ofdepth map capture techniques may be employed on a given camera rig, anysuitable combination or configuration of depth map capture techniquesmay also be employed on a given node (e.g., such as nodes 302 of FIG. 3,nodes 504 of FIG. 5, etc.) as may serve a particular implementation.

Depth map capture technique 700-A and variants of depth map capturetechnique 700-A may be referred to herein as structured light depth mapcapture techniques. Structured light depth map capture technique 700-Ais illustrated in FIG. 7A by a top view of various elements of a depthmap capture subsystem along with object 202, which was described above.Specifically, as shown, the depth map capture subsystem performingstructured light depth map capture technique 700-A may include astructured light emitter 702 disposed at a first fixed position withrespect to the real-world scene and configured to emit a structuredlight pattern onto the surfaces of the objects included in thereal-world scene (e.g., object 202). The depth map capture subsystemperforming structured light depth map capture technique 700-A may alsoinclude a structured light sensor 704 disposed at a second fixedposition with respect to the real-world scene and configured to detectthe structured light pattern as the structured light pattern reflectsoff of the surfaces of the objects included in the real-world scene.While structured light emitter 702 and structured light sensor 704 areillustrated in FIG. 7A as slightly converging (e.g., pointing inwardtoward one another), it will be understood that in certainimplementations, structured light emitter 702 and structured lightsensor 704 may be oriented at any suitable angle with respect to oneanother and/or with respect to the real-world scene. For example,structured light emitter 702 and structured light sensor 704 may beoriented parallel to one another (i.e., both pointing straight aheadrather than pointing inward) or even oriented to be pointing somewhatoutward from one another as may serve a particular implementation.

The depth map capture subsystem performing structured light depth mapcapture technique 700-A may capture a depth map by generating the depthmap based on the detected structured light pattern and the first andsecond fixed positions. For example, due to the shape of object 202, thestructured light pattern emitted by structured light emitter 702 mayfollow a curvature of the surfaces of object 202 and may appear, fromthe fixed position of structured light sensor 704 with respect to object202, to bend and contour to the surfaces of object 202. As such, thedepth map capture subsystem may determine depth data representative ofthe surfaces of object 202 by triangulating particular physical pointson surfaces of object 202 based on known angles and fixed positions ofboth structured light emitter 702 and structured light sensor 704, andbased on how particular light associated with the structured lightpattern reflects from the particular physical points on the surfaces ofobject 202.

Structured light depth map capture technique 700-A may be used toeffectively capture depth data for surfaces that are smooth andrelatively featureless (e.g., certain types of walls, ceilings, orfloors, objects having smooth surfaces such as object 202, etc).Additionally, structured light depth map capture technique 700-A may beused to effectively capture depth data for surfaces of objects that arerelatively near to or far away from structured light emitter 702 and/orstructured light sensor 704. However, structured light depth map capturetechnique 700-A may be less effective when used to capture depth datafrom surfaces that absorb light at the frequency that the structuredlight pattern is emitted since the structured light pattern may not beeffectively reflected to be detected by structured light sensor 704.Additionally, structured light depth map capture technique 700-A may bemost effective within certain areas of a structured light pattern (e.g.,areas near the middle of the structured light pattern) while being lesseffective at other areas (e.g., near the edges of the structured lightpattern).

Structured light emitter 702 may include any suitable hardware orcombination of hardware and software configured to emit a structuredlight pattern onto the surfaces of object 202. For example, structuredlight emitter 702 may include or be implemented by any of the componentsdescribed above in relation to depth map capture subsystem 204 or depthmap access facility 102 of system 100. In certain implementations,structured light emitter 702 may include a device configured to generatestimulated emission of electromagnetic radiation (e.g., a visible lightlaser or equivalent IR device or other device associated with anotherpart of the electromagnetic spectrum) to emit a light beam 706 that maybe processed, split, shaped, filtered, or otherwise treated by anoptical element 708. For example, optical element optical element 708may include or be implemented by one or more lenses, mirrors, filters,splitters, combiners, prisms, or other diffractive optical elements, ora series thereof, that may be configured to split and/or shape lightbeam 706 into a structured light pattern that is emitted (i.e.,projected, shined, etc.) onto object 202 and/or other objects in thevicinity of object 202 within a real-world scene (not explicitly shown).

Optical element 708 may split, shape, or otherwise alter light beam 706to form the structured light pattern in any way as may serve aparticular implementation. In certain examples, optical element may beoptional and structured light emitter 702 may emit the structured lightpattern without using light beam 706 and/or optical element 708.

The structured light pattern emitted by structured light emitter 702 mayinclude or be implemented by any suitable pattern of light (e.g., IRlight, visible light, etc.), as may be emitted by structured lightemitter 702. Arrows shown to be emanating from structured light emitter702 in FIG. 7A represent the structured light pattern by illustrating,from the top view, a sector (e.g., a sector of the real-world scene)that may be illuminated by the structured light pattern based on theposition, projection angle, and other characteristics of structuredlight emitter 702. The pattern of the structured light is not shown inFIG. 7A due to the geometry of the top view, but it will be understoodthat the structured light pattern is projected onto the surfaces ofobject 202.

The structured light pattern may be patterned in any suitable way, suchas, for example, with a pattern of dots that are uniformly sized and/oruniformly distributed, a pattern of dots with varied sizes and/or varieddistribution patterns to facilitate identification of particular dotswithin the structured light pattern, a pattern of dots with random sizesand random distribution across the surfaces of the objects in thereal-world scene, a pattern of stripes (e.g., horizontal, vertical, ordiagonal stripes with uniform, varied, or random stripe widths and/ordistributions), a checkered pattern, and/or any other suitable patternof structured light. In certain examples where multiple structured lightpatterns overlap on the surface of an object, different patterns (e.g.,complementary patterns that facilitate depth capture of differentsurfaces or different features of the surfaces) may be used.

When light included within the structured light pattern reaches thesurfaces of the objects in the real-world scene (e.g., such as object202), the light may reflect from the surfaces and travel back towardstructured light emitter 702 and structured light sensor 704. As such,FIG. 7A shows arrows pointing toward structured light sensor 704 toillustrate a structured light pattern reflection originating from (i.e.,reflecting off of) object 202 (e.g., and/or other objects within thereal-world scene not explicitly shown).

Structured light sensor 704 may include any suitable hardware orcombination of hardware and software configured to detect (e.g., sense,receive, record, etc.) a structured light pattern reflecting from thesurfaces of object 202. For example, structured light sensor 704 mayinclude or be implemented by any of the components described above inrelation to depth map capture subsystem 204 or depth map convergencefacility 104 of system 100 as well as other components (e.g., optics,filters, etc.) as may serve a particular implementation.

Structured light sensor 704 may be sensitive to light at a particularrange of frequencies including a frequency or range of frequencies atwhich the structured light pattern is emitted by structured lightemitter 702. For example, structured light sensor 704 may include or beimplemented by a video camera that is sensitive to light at frequencieswithin the range of visible light. As another example, structured lightsensor 704 may include or be implemented by an IR sensor that issensitive to light in an IR portion of the electromagnetic spectrum. Forexample, the IR sensor may be sensitive to all or virtually allfrequencies in an IR portion of the electromagnetic spectrum,frequencies in a “near IR” portion of the electromagnetic spectrum, IRfrequencies associated with wavelengths from approximately 600nanometers (“nm”) to approximately 1000 nm, IR frequencies associatedwith wavelengths from approximately 700 nm to approximately 950 nm, IRfrequencies associated with wavelengths from approximately 750 nm toapproximately 875 nm, or any other frequency range as may serve aparticular implementation.

Depth map capture technique 700-B and variants of depth map capturetechnique 700-B may be referred to herein as stereoscopic depth mapcapture techniques. As with structured light depth map capture technique700-A, stereoscopic depth map capture technique 700-B is illustrated inFIG. 7B by a top view of various elements of a depth map capturesubsystem along with object 202. Specifically, as shown, the depth mapcapture subsystem performing stereoscopic depth map capture technique700-B may include a first optical sensor 710-1 disposed at a first fixedposition with respect to the real-world scene and configured to detectthe surfaces of the objects included in the real-world scene (e.g.,object 202) from the first fixed position. The depth map capturesubsystem performing stereoscopic depth map capture technique 700-B mayalso include a second optical sensor 710-2 disposed at a second fixedposition (e.g., a fixed position offset from the first fixed position bya predetermined distance) with respect to the real-world scene andsimilarly configured to detect the surfaces of the objects included inthe real-world scene from the second fixed position. While opticalsensors 710-1 and 710-2 are illustrated in FIG. 7B as slightlyconverging (e.g., pointing inward toward one another), it will beunderstood that in certain implementations, optical sensors 710 may beoriented at any suitable angle with respect to one another and/or withrespect to the real-world scene. For example, optical sensors 710 may beoriented parallel to one another (i.e., both pointing straight aheadrather than pointing inward) or even oriented to be pointing somewhatoutward from one another as may serve a particular implementation.

The depth map capture subsystem performing stereoscopic depth mapcapture technique 700-B may capture a depth map by generating the depthmap based on stereoscopic differences in how the surfaces of the objectsare detected from the first and second fixed positions. For example,based on stereoscopic differences in how particular physical points onsurfaces of object 202 appear from the first fixed position and how theparticular physical points appear from the second fixed position, thedepth map capture subsystem may triangulate the particular physicalpoints in space based on known angles and fixed positions of bothoptical sensors 710 (i.e., first optical sensor 710-1 and second opticalsensor 710-2) to determine depth data for the particular physicalpoints.

Stereoscopic depth map capture technique 700-B may be used toeffectively capture depth data for textured surfaces or surfaces ofobjects that include lots of edges, corners, and other easilyidentifiable features. Additionally, stereoscopic depth map capturetechnique 700-B may be used to effectively capture depth data forsurfaces that may absorb light at various specific frequencies (e.g.,specific frequencies at which a structured light pattern may be emitted)since the surfaces also likely reflect light at other frequencies thatmay be detected by optical sensors 710. However, stereoscopic depth mapcapture technique 700-B may be less effective at detecting depth datafor surfaces that are smooth and relatively featureless or for surfacesof objects that are relatively far away from optical sensors 710.Additionally, stereoscopic depth map capture technique 700-B may be lesseffective at detecting depth data for surfaces of objects that are notnaturally well lit by ambient light (e.g., surfaces that are in shadowsor other dark areas, etc.).

As mentioned above, because structured light depth map capture technique700-A is effective at capturing depth data for surfaces thatstereoscopic depth map capture technique 700-B is not (e.g., smoothsurfaces, far away surfaces, poorly-lit surfaces, etc.) and stereoscopicdepth map capture technique 700-B is effective at capturing depth datafor surfaces that structured light depth map capture technique 700-A isnot (e.g., surfaces that absorb particular frequencies at which astructured light pattern is emitted, close-by surfaces that areilluminated by an edge of a structured light pattern, etc.), depth mapcapture techniques 700-A and 700-B may be complementary depth mapcapture techniques.

Optical sensors 710 may include any suitable hardware or combination ofhardware and software configured to detect (e.g., sense, receive,record, etc.) light reflecting from the surfaces of object 202. Forexample, ambient light present within the real-world scene may reflectfrom the surfaces of the objects in the real-world scene (e.g., such asobject 202) and travel toward optical sensors 710 at their respectivefixed positions. As such, FIG. 7B shows arrows pointing toward eachrespective optical sensor 710 to illustrate light (e.g., ambient light)reflecting from object 202 (e.g., and/or other objects within thereal-world scene not explicitly shown).

Optical sensors 710 may include or be implemented by any of thecomponents described above in relation to depth map capture subsystem204 or depth map convergence facility 104 of system 100 as well as othercomponents (e.g., optics, filters, etc.) as may serve a particularimplementation. For example, optical sensors 710 may be implemented byvideo cameras that are sensitive to visible light and/or light at otherfrequencies on the electromagnetic spectrum. In some examples, opticalsensors 710 may be similar to or equivalent to structured light sensor704.

Depth map capture technique 700-C and variants of depth map capturetechnique 700-C may be referred to herein as time-of-flight depth mapcapture techniques. As with depth map capture techniques 700-A and700-B, time-of-flight depth map capture technique 700-C is illustratedin FIG. 7C by a top view of various elements of a depth map capturesubsystem along with object 202. Specifically, as shown, the depth mapcapture subsystem performing time-of-flight depth map capture technique700-C may include a light pulse source 712 configured to emit a lightpulse at a particular time and a time-of-flight sensor 714 configured todetect, at a plurality of different times subsequent to the particulartime, the emitted light pulse as the emitted light pulse reflects off ofthe surfaces of the objects included in the real-world scene (e.g.,including object 202) in accordance with the respective depths of thesurfaces of the objects. While light pulse source 712 and time-of-flightsensor 714 are illustrated in FIG. 7C as being positioned directly ontop of one another (e.g., pointing in a parallel direction), it will beunderstood that in certain implementations, light pulse source 712 andtime-of-flight sensor 714 may be oriented at any suitable angle withrespect to one another and/or with respect to the real-world scene. Forexample, light pulse source 712 and time-of-flight sensor 714 may beoriented to be pointing inward toward one another or outward away fromone another as may serve a particular implementation.

The depth map capture subsystem performing time-of-flight depth mapcapture technique 700-C may capture a depth map by generating the depthmap based on the different times within the plurality of different timessubsequent to the particular time at which the emitted light pulse isdetected. For example, as illustrated by the time graph shown under theillustration of depth map capture technique 700-C in FIG. 7C, a lightpulse configured to bathe the real-world scene in light (i.e., toreflect off all the surfaces of the objects visible from the perspectiveof light pulse source 712 like a camera flash) may be emitted at aparticular time 716 that represents a singular moment in time or arelatively short period of time. Because light from the emitted lightpulse travels from light pulse source 712 to the surfaces of the objectswithin the real-world scene (e.g., the surfaces of object 202) and then,after being reflected from the surfaces, travels back to be detected attime-of-flight sensor 714 with a constant and predictable speed,different arrival times 718 of light reflecting from different physicalpoints on the surfaces of the objects may indicate the respective depthsof the different physical points on the surfaces. Accordingly, bytracking and correlating precise arrival times 718 of light reflectingback from the surfaces of object 202, for example, the depth map capturesubsystem may determine depth data representative of the surfaces ofobject 202.

Time-of-flight depth map capture technique 700-C may have certaincharacteristics in common with both structured light depth map capturetechnique 700-A (e.g., it may rely on projected light rather thanambient light, etc.) and stereoscopic depth map capture technique 700-B(e.g., it may detect multi-frequency light that bathes a real-worldscene rather than light emitted at a particular frequency only toparticular areas of the real-world scene according to a pattern, etc.).As such, time-of-flight depth map capture technique 700-C may be used toeffectively capture depth data for certain types of surfaces while beingless effective with other types of surfaces in accordance with theseand/or other characteristics as described above in relation to depth mapcapture techniques 700-A and 700-B. Time-of-flight depth map capturetechnique 700-C may thus be complementary in at least certain respectswith both structured light depth map capture technique 700-A andstereoscopic depth map capture technique 700-B.

Light pulse source 712 and time-of-flight sensor 714 may each includeany suitable hardware or combination of hardware and software configuredto emit a light pulse and/or detect (e.g., sense, receive, record, etc.)light reflecting from the surfaces of object 202. For example, lightpulse source 712 may emit light at a particular frequency that is notpresent or prevalent in ambient light present at the real-world sceneand time-of-flight sensor 714 may detect light at the particularfrequency (i.e., so as to distinguish light from an emitted light pulsefrom other ambient light present at the real-world scene). In someexamples, light pulse source 712 and time-of-flight sensor 714 may beintegrated into a single device or may be included within separatedevices as may serve a particular implementation. As shown, fixedpositions of light pulse source 712 and of time-of-flight sensor 714with respect to the real-world scene may be related (e.g., closelyaligned so as to be equivalent for the purposes of time-of-flight depthmap capture technique 700-C). As such, FIG. 7C shows arrows pointingaway from light pulse source 712 to illustrate the light pulse bathing asector of the real-world scene in light from the fixed position of lightpulse source 712 and arrows pointing toward time-of-flight sensor 714 toillustrate light from the emitted light pulse reflecting from thesurfaces of object 202 (e.g., and/or other surfaces of other objectswithin the real-world scene not explicitly shown) to be detected attime-of-flight sensor 714.

It will be understood that depth map capture techniques 700-A through700-C illustrated in FIGS. 7A-7C are exemplary only and that variationsof depth map capture techniques 700-A through 700-C and/or additionaldepth map capture techniques not explicitly illustrated may also be usedto independently capture depth maps in certain implementations. Forexample, a stereo-disparity depth map capture technique (e.g., aninfrared stereo-disparity depth map capture technique) may be employedin certain implementations. A stereo-disparity depth map capturetechnique may be understood to be a variant of either or both ofstructured light depth map capture technique 700-A and stereoscopicdepth map capture technique 700-B, and may share certain characteristicswith both of these depth map capture techniques.

Specifically, similar to structured light depth map capture technique700-A, the stereo-disparity depth map capture technique may projectstructured light of a particular frequency (e.g., an infrared frequency)onto a real-world scene using a diffractive optical element such asthose described herein. For example, the structured light pattern mayinclude a field of random dots. Then, similar to stereoscopic depth mapcapture technique 700-B, the infrared stereo-disparity depth map capturetechnique may include optical sensors (e.g., monochrome cameras and/orany other optical sensor described herein) arranged in a stereo pairarrangement similar to optical sensors 710-1 in depth map capturetechnique 700-B, but with infrared bandpass filters that eliminatevisible scene illumination. In this way, data detected by the opticalsensors may be processed to generate a depth map based on the projectedstructured light pattern and calculated based on stereo-disparityalgorithms such as those described above with respect to depth mapcapture technique 700-B. Accordingly, for example, the stereo-disparitydepth map capture technique may include an structured light emitter(e.g., an infrared structured light emitter) with an associateddiffractive optical element, two monochrome cameras, and a color camera(e.g., having similar resolution to the monochrome cameras) to be usedin detecting full color textures of objects within the real-world scene.

Various implementations of depth map capture subsystems included withinsystem 100 have been described and illustrated. For example, variousprinciples of depth data capture and depth map capture subsystems weredescribed in relation to FIG. 2A, various aspects of the positioning ofdepth map capture subsystems with respect to a real-world scene (e.g.,via nodes, camera rigs, etc.) were described in relation to FIGS. 3through 5, and various depth map capture techniques used for capturingindependent depth maps were described in relation to FIGS. 7A-7C. Once aplurality of depth map capture subsystems (e.g., included within orcommunicatively coupled to system 100) have been configured to performappropriate depth map capture techniques and have been appropriatelydisposed at fixed positions with respect to a real-world scene inaccordance with the principles described above, the depth map capturesubsystems may independently capture depth maps to be converged bysystem 100 into a converged depth map of the surfaces of the objectsincluded in the real-world scene. To illustrate, various aspects of thecapturing and converging of independently-captured depth maps will nowbe described in relation to FIGS. 8A-8C and 9.

FIG. 8A illustrates exemplary components of an exemplary implementation800 of system 100 generating depth data by convergingindependently-captured depth maps. More specifically, implementation 800shows two depth map capture subsystems 802 (i.e., depth map capturesubsystems 802-1 and 802-2) each associated with a respective scope ofcapture 804 (i.e., scope of capture 804-1 associated with depth mapcapture subsystem 802-1 and scope of capture 804-2 associated with depthmap capture subsystem 802-2) that includes object 202. Depth map capturesubsystems 802 may be similar or the same as other depth map capturesubsystems described herein and each respective scope of capture 804 mayinclude, along with object 202, other objects within a real-world sceneor a portion of a real-world scene as may serve a particularimplementation.

Although not explicitly shown in FIG. 8A for clarity, it will beunderstood that implementation 800 of system 100 may further includeadditional or fewer components such as those described with respect toother implementations of system 100 described herein. For example, asmentioned above, implementation 800 may include one or more additionaldepth map capture subsystems positioned at additional fixed positionswith respect to the real-world scene (e.g., at positions withindifferent camera rigs of a same node, at positions within differentnodes at different fixed node positions, etc.). Moreover, implementation800 may include computing resources (e.g., servers or other computingdevices included in or implementing a data capture processing unit suchas data capture processing unit 310), and/or any other componentsdescribed herein or as may serve a particular implementation. Similarly,while implementation 800 is shown to include depth map capturesubsystems 802 and to access depth maps by capturing the depth maps, itwill be understood that, in some examples, implementation 800 of system100 may not include depth map capture subsystems 802 but, rather, mayinclude only a computing system configured to accessindependently-captured depth maps by receiving the depth maps from depthmap capture subsystems 802 as depth map capture subsystems 802 capturethe depth maps.

As described above, depth maps captured by each of depth map capturesubsystems 802 may be independent. As a first example, implementation800 may include a plurality of nodes each disposed at a different fixednode position in a plurality of fixed node positions with respect to thereal-world scene and each depth map capture subsystem 802 may capture adepth map from a different node in the plurality of nodes. In thisexample, the depth maps may be independently captured even if both depthmap capture subsystems 802 capture the respective depth maps using thesame depth map capture technique. For example, both depth map capturesubsystems may capture the respective depth map using a structured lightdepth map capture technique, a stereoscopic depth map capture technique,a time-of-flight depth map capture technique, or any other depth mapcapture technique as may serve a particular implementation.

As a second example, each depth map capture subsystem 802 may capture arespective depth map by way of a different depth map capture technique.For example, depth map capture subsystem 802-1 may capture a first depthmap by way of a first depth map capture technique selected from thestructured light depth map capture technique, the stereoscopic depth mapcapture technique, the time-of-flight depth map capture technique, andother depth map capture techniques that may serve a particularimplementation, while depth map capture subsystem 802-2 may capture asecond depth map by way of a second, different depth map capturetechnique selected from the same list of depth map capture techniques.The first and second depth maps may be independently captured even if,for example, implementation 800 includes a plurality of nodes eachdisposed at a different fixed node position in a plurality of fixed nodepositions with respect to the real-world scene and the capturing of thefirst depth map and the capturing of the second depth map are bothperformed from a same particular node (e.g., and/or a same particularcamera rig within the particular node) in the plurality of nodes.

Once independently-captured depth maps have been captured and/orotherwise accessed by system 100 (e.g., by being captured by depth mapcapture subsystems 802), system 100 may perform certain operations toconverge the independently-captured depth maps into a converged depthmap. For example, system 100 may assign respective confidence values toeach depth data point in a first plurality of depth data points of thefirst depth map (i.e., the depth map captured by depth map capturesubsystem 802-1) and assign other respective confidence values to eachdepth data point in a second plurality of depth data points of thesecond depth map (i.e., the depth map captured by depth map capturesubsystem 802-2).

As illustrated in FIGS. 8B and 8C respectively, perspective views 806-1and 806-2 of object 202 (i.e., collectively referred to herein asperspective views 806) illustrate object 202 from the perspective ofeach respective depth map capture subsystem 802 (i.e., view 806-1illustrating object 202 from the perspective of depth map capturesubsystem 802-1 and view 806-2 illustrating object 202 from theperspective of depth map capture subsystem 802-2). As shown, varioussurfaces of object 202 (i.e., surfaces 210-1 through 210-5) are shown ineach of views 806, although it will be understood that other surfaces ofobject 202 may also exist that may not be visible from the perspectiveof either of views 806. As described above with respect to FIG. 2B, eachsurface 210 may include a plurality of physical points for which depthdata may be captured by depth map capture subsystem 204, including, forexample, physical point 212.

Confidence values may be assigned to depth data points representative ofeach physical point on object 202 (e.g., including physical point 212).In certain examples, as part of or in preparation for assigningconfidence values to depth data points, a calibration process may beperformed to convert depth data detected by each of depth map capturesubsystems 802 from a local coordinate system used only by therespective depth map capture subsystem to a universal or “world”coordinate system shared by all the depth map capture subsystemsincluded within system 100. For example, by way of the calibrationprocess, each depth map capture subsystem 802 may be assigned atransformation matrix by which data captured by the depth map capturesubsystem 802 may be compared, rotated, and/or otherwise aligned withdata captured by other depth map capture subsystems 802. Based on therespective transformation matrices and the apparent positions of variousphysical points with respect to the world coordinate system, one depthmap capture subsystem 802 (e.g., depth map capture subsystem 802-1) maycorrelate a particular depth data point (e.g., a depth data pointrepresentative of physical point 212) with a particular depth data pointrepresentative of a same physical point (i.e., physical point 212)captured by the other depth map capture subsystem 802 (i.e., depth mapcapture subsystem 802-2).

Once a first depth data point is correlated with a second depth datapoint corresponding to the same physical point in this manner, aconfidence value may be assigned to each depth data point, and the depthdata points may be converged according to the confidence values toattempt to generate a depth data point corresponding to the samephysical point that may be, for example, at least as accurate as or moreaccurate than either the first or second depth data points. An exampleof how confidence values may be assigned and converged will now beprovided. For clarity of description, the example will illustrate howconfidence values may be assigned and converged with respect torespective depth data points captured by different depth map capturesubsystems 802 but representing the same physical point (i.e., physicalpoint 212 on surface 210-4). However, it will be understood that theprinciples illustrated by the following example may apply to other depthdata points representing other physical points on surface 210-4, onother surfaces 210 of object 202, or on other surfaces of other objectswithin a real-world scene.

First, the assigning of a first confidence value to a first depth datapoint (e.g., the depth data point captured by depth map capturesubsystem 802-1 for physical point 212) may be performed. For example,the first confidence value may be assigned based on at least one of anattribute of the particular surface that includes physical point 212(i.e., surface 210-4) and a first viewing perspective (e.g., withrespect to surface 210-4) of a component of depth map capture subsystem802-1 capturing the first depth map (e.g., a structured light sensor, anoptical sensor, a time-of-flight sensor, etc., based on the depth mapcapture technique used). Second, the assigning of the second confidencevalue to a second depth data point (e.g., the depth data point capturedby depth map capture subsystem 802-2 for physical point 212) may beperformed in a similar way. Specifically, the second confidence valuemay be assigned based on at least one of an attribute of surface 210-4(e.g., the same or a different attribute upon which the first confidencevalue was based) and a second viewing perspective (e.g., with respect tosurface 210-4) of a component of depth map capture subsystem 802-2capturing the second depth map.

Confidence values may be assigned based on any suitable attribute of asurface. For example, system 100 may determine that one attribute ofsurface 210-4 (of which physical point 212 is a part) is that surface210-4 is relatively smooth. Accordingly, since structured light depthmap capture techniques may be more effective with smooth surfaces than,for example, stereoscopic depth map capture techniques (as describedabove), system 100 may be more likely to assign a higher confidencevalue to a depth data point representative of physical point 212 (orother physical points on surface 210-4) if a structured light depth mapcapture technique is used to capture the depth data point, than if, forexample, a stereoscopic depth map capture technique is used to capturethe depth data point.

Similarly, confidence values may be assigned based on other attributesof surfaces such as how textured or non-textured (e.g., smooth,featureless, etc.) the surfaces are, how near or far the surfaces arefrom the depth map capture subsystems, how well or poorly lit thesurfaces are, the degree to which the surfaces absorb or reflect lightat certain frequencies (e.g., a particular frequency emitted for astructured light pattern), and various other attributes as may serve aparticular implementation.

Attributes may be determined for each physical point on a particularsurface based on attributes of physical points surrounding the physicalpoint on the particular surface, attributes of physical points definingthe particular surface (e.g., physical points at corners or edges of thesurface), or by any other suitable physical points detected by the depthmap capture subsystems and/or using any suitable algorithms ortechniques for determining attributes of object surfaces in an image(e.g., computer vision techniques and/or algorithms for extractingfeatures such as graph cuts, belief propagation, extrapolation fromHarris corners, etc.). For example, a surface may be determined to besmooth if a physical point on the surface is surrounded by various otherphysical points that look very similar to the physical point such that aderivative from point to point shows little change between points, whilea surface may be determined to be more textured if the derivative showsgreater or more abrupt changes from point to point.

Confidence values may also be assigned based on a viewing perspective(e.g., with respect to a surface upon which a physical point isincluded) of the sensing component (e.g., structured light sensor,optical sensor, time-of-flight sensor, etc.) of the depth map capturesubsystem 802 in any way as may serve a particular implementation. Forexample, as illustrated in view 806-1, system 100 may determine thatrelatively effective (e.g., accurate) depth data may be captured forphysical point 212 based on the perspective of the sensing component ofdepth map capture subsystem 802-1 because surface 210-4 can be analyzedfairly straight-on from the perspective of depth map capture subsystem802-1. Accordingly, system 100 may assign a relatively high confidencevalue to a depth data point captured by depth map capture subsystem802-1. Conversely, as illustrated in view 806-2, system 100 maydetermine that depth data for physical point 212 may be unlikely to beparticularly effective or accurate based on the perspective of thesensing component of depth map capture subsystem 802-2 because surface210-4 must be analyzed from a fairly sharp angle from the perspective ofdepth map capture subsystem 802-2, making the depth data of physicalpoint 212 more difficult to correctly capture.

Various other factors may also be used alone or in conjunction witheither or both of the factors described above (i.e., the attributes of asurface and the perspective or viewing angle that a depth map capturesubsystem has with respective to the surface) to assign confidencevalues. For example, if physical point 212 is captured by multiple depthmap capture subsystems associated with one node (e.g., multiple depthmap capture subsystems on multiple camera rigs), a confidence valueassigned for a depth map associated with the entire node (e.g., formedfrom converging one or more depth maps associated with the multiplecamera rigs) may be higher than if physical point 212 is only capturedby one depth map capture subsystem associated with the node. Similarly,other factors may be used to assign confidence values as may serve aparticular implementation.

Once system 100 accesses the independently-captured depth maps andassigns confidence values to depth data points within the depth maps,system 100 may generate a converged depth map by converging depth datapoints included in the independently-captured depth maps based on theassigned confidence values in any suitable way. For instance, tocontinue with the example of physical point 212 described above, system100 may generate a third, converged depth data point from the first andsecond depth data points captured by each of depth map capturesubsystems 802 by assigning to the third depth data point whichever ofthe first and second depth data points have been assigned a higherconfidence value. More specifically, system 100 may generate the thirddepth data point by identifying a maximum confidence value from thefirst and second confidence values and, if the maximum confidence valueis the first confidence value, assigning the first depth data point tothe third depth data point, and, if the maximum confidence value is thesecond confidence value, assigning the second depth data point to thethird depth data point. If the confidence values are equal in thisexample, system 100 may choose the first or second depth data point toassign to the third depth data point, average the depth data point, orotherwise break the tie in any way as may serve a particularimplementation.

Alternatively, system 100 may generate the third depth data point as aweighted average of the first and second depth data points in accordancewith the first and second confidence values. For example, system 100 maymathematically determine, and assign to the third depth data point, adepth data point between the first and second depth data points that iscloser to one depth data point or the other based on how high theconfidence value is for each depth data point. In other words, if thefirst depth data point has been assigned a relatively high confidencevalue and the second depth data point has been assigned a relatively lowconfidence value, system 100 may generate a third depth data pointbetween the first and second depth data points that is closer to thefirst depth data point than to the second depth data point due to thehigher confidence value assigned to the first depth data point.

To illustrate, FIG. 9 illustrates an exemplary converging 900 of twoindependently-captured depth maps 902 (i.e., depth maps 902-1 and 902-2)into exemplary converged depth maps 904 (i.e., converged depth maps904-1 and 904-2). The depth maps illustrated in FIG. 9 (i.e., depth maps902, as well as converged depth maps 904) illustrate depth data undercolumns labeled “Depth” for a relatively small number of depth datapoints under columns labeled “Point.” While the depth maps of FIG. 9 arerelatively simple (e.g., including only one-dimensional depth data foronly three depth data points) for clarity of illustration anddescription, it will be understood that depth maps generated by system100 may be considerably more complex and/or comprehensive. For example,depth maps generated by system 100 may include hundreds, thousands, oreven more depth data points, each of which may be associated withvarious dimensions of depth data. For instance, in some implementations,three dimensions of depth data (e.g., corresponding to orthogonal x, y,and z coordinates) may be captured and associated with each depth datapoint. While converging 900 illustrates two independently-captured depthmaps being converged to form a converged depth map, the principlesdescribed may similarly apply to an arbitrary number ofindependently-captured depth maps (e.g., more than two depth maps) beingconverged to form the converged depth map.

Depth maps 902 may be independently captured by way of any of the depthmap capture techniques and/or from any of the fixed positions orconfigurations described herein. Each depth data point (i.e., A, B, C)represented in depth maps 902 may correspond to a particular physicalpoint included on a surface of an object in a real-world scene. Forexample, depth data point A may correspond with physical point 212,illustrated in FIGS. 2 and 8, while depth data points B and C maycorrespond with other physical points on surfaces 210 of object 202 orwith physical points on other surfaces of other objects within thereal-world scene in which object 202 is included. Regardless, due toalignment and calibration performed as described above, depth data pointA in depth map 902-1 may be representative of the same physical point asdepth data point A in depth map 902-2 and depth data point A in both ofconverged depth maps 904, and so forth.

Along with columns identifying the depth data point and thecorresponding depth that has been captured for each depth data point,depth maps 902 further include columns labeled “Confidence” in which anassigned confidence value for each depth data point is shown. In someexamples, the confidence value for each depth data point may be assignedbased on a single scale and/or paradigm (e.g., confidence percentagevalues, raw confidence values based on a particular scale, binarypass/fail-type confidence values, etc.). However, to illustratedifferent paradigms that may be used, each depth data point in FIG. 9 isassigned a different type of confidence value. Specifically, as shown indepth maps 902, depth data point A is assigned a confidence percentagevalue (i.e., 71% in the case of depth map 902-1 and 53% in the case ofdepth map 902-2), depth data point B is assigned a raw confidence valuebased on a particular scale such as a scale from 0 to 499 (i.e., 135 inthe case of depth map 902-1 and 313 in the case of depth map 902-2), anddepth data point C is assigned a binary (e.g., pass/fail-type)confidence value (i.e., N to represent “No” confidence in the case ofdepth map 902-1 and “Y” to represent “Yes” confidence in the case ofdepth map 902-2).

Converged depth maps 904 illustrate two alternative converged depth mapsthat system 100 may generate from depth maps 902 based on how system 100is configured to perform the converging of the depth maps. Specifically,converged depth map 904-1 illustrates an example where system 100assigns to each depth data point in converged depth map 904-1 whicheverof the respective first and second depth data points have been assigneda higher confidence value. As shown, because depth map 902-1 has ahigher confidence value for depth data point A (i.e., 71% rather than53%), depth data point A in converged depth map 904-1 is assigned 57(i.e., the depth data associated with depth data point A in depth map902-1). Because depth map 902-2 has a higher confidence value for depthdata point B (i.e., 313 rather than 135), depth data point B inconverged depth map 904-1 is assigned 28 (i.e., the depth dataassociated with depth data point B in depth map 902-2). Similarly,because depth map 902-2 has a higher confidence value for depth datapoint C (i.e., Y rather than N), depth data point C in converged depthmap 904-1 is assigned 52 (i.e., the depth data associated with depthdata point C in depth map 902-2).

As an alternative example, converged depth map 904-2 illustrates anexample where system 100 assigns to each depth data point in convergeddepth map 904-2 a weighted average of the first and second depth datapoints in accordance with the first and second confidence values indepth maps 902. Specifically, as shown, system 100 assigns 54.38 todepth data point A in converged depth map 904-2 based on a weightedcalculation of the respective depth data values captured by depth maps902 (i.e., 57 and 51, respectively) that takes into account therespective confidence values associated with each depth data value.Specifically, the weighting results in the converged depth data fordepth data point A (i.e., 54.38) being closer to a depth data value of57 (i.e., the depth data value for depth data point A in depth map 902-1which is associated with a higher confidence value of 71%) than to adepth data value of 51 (i.e., the depth data value for depth data pointA in depth map 902-2, which is associated with a lower confidence valueof 53%). Similarly, the converged depth data value for depth data pointB is 26.19, which is closer to the depth data value of 28 in depth map902-2 than to the depth data value of 22 in depth map 902-1 due to thehigher confidence value of depth map 902-2. Because depth map 902-1 hasno confidence for depth data point C, the converged depth data value fordepth data point C in converged data map 904-2 may be 52 to reflect theonly depth data value for which sufficient confidence is indicated.

It will be understood that variants of these techniques for convergingdepth data from depth maps 902-1 and 902-2 into depth data of aconverged depth map 904, a combination of these techniques, or othersuitable techniques different from these techniques may be employed incertain examples as may serve a particular implementation.

FIG. 10 illustrates an exemplary method 1000 for generating depth databy converging independently-captured depth maps. While FIG. 10illustrates exemplary operations according to one embodiment, otherembodiments may omit, add to, reorder, and/or modify any of theoperations shown in FIG. 10. One or more of the operations shown in FIG.10 may be performed by system 100 and/or any implementation thereof.

In operation 1002, a depth data generation system may access a firstdepth map of surfaces of objects included in a real-world scene. Forexample, the first depth map may include a first plurality of depth datapoints each representative of a different physical point included in aplurality of physical points on the surfaces of the objects included inthe real-world scene. Operation 1002 may be performed in any of the waysdescribed herein.

In operation 1004, the depth data generation system may access a seconddepth map of the surfaces of the objects included in the real-worldscene. The second depth map accessed in operation 1004 may have beencaptured independently from the first depth map accessed in operation1002. Additionally, the second depth map may include a second pluralityof depth data points each representative of a different physical pointincluded in the plurality of physical points on the surfaces of theobjects included in the real-world scene. Operation 1004 may beperformed in any of the ways described herein.

In operation 1006, the depth data generation system may converge thefirst and second depth maps into a converged depth map of the surfacesof the objects included in the real-world scene. For example, theconverged depth map may include a third plurality of depth data pointseach representative of a different physical point included in theplurality of physical points on the surfaces of the objects included inthe real-world scene. Operation 1006 may be performed in any of the waysdescribed herein. For example, as illustrated in FIG. 10, operation 1006may be performed by performing operations 1008 through 1012, describedbelow.

In operation 1008, the depth data generation system may assign a firstconfidence value to a first depth data point in the first plurality ofdepth data points of the first depth map. For example, the first depthdata point may represent a particular physical point included in theplurality of physical points. Operation 1008 may be performed in any ofthe ways described herein.

In operation 1010, the depth data generation system may assign a secondconfidence value to a second depth data point in the second plurality ofdepth data points of the second depth map. In some examples, the seconddepth data point, like the first depth data point, may also representthe particular physical point. Operation 1010 may be performed in any ofthe ways described herein.

In operation 1012, the depth data generation system may generate a thirddepth data point in the third plurality of depth data points. The thirddepth data point may also represent the particular physical point.Operation 1012 may be performed in any of the ways described herein. Forexample, the third depth data point may be generated based on the firstand second confidence values and on at least one of the first depth datapoint in the first plurality of depth data points and the second depthdata point in the second plurality of depth data points.

FIG. 11 illustrates an exemplary method 1100 for generating depth databy converging independently-captured depth maps. While FIG. 11illustrates exemplary operations according to one embodiment, otherembodiments may omit, add to, reorder, and/or modify any of theoperations shown in FIG. 11. One or more of the operations shown in FIG.11 may be performed by system 100 and/or any implementation thereof.

In operation 1102, a depth data generation system may capture a firstdepth map of surfaces of objects included in a real-world scene. Forexample, the first depth map may include a first plurality of depth datapoints each representative of a different physical point included in aplurality of physical points on the surfaces of the objects included inthe real-world scene. Operation 1102 may be performed in any of the waysdescribed herein. For example, the first depth map may be captured froma particular node disposed at a particular fixed node position withrespect to the real-world scene and using a structured light depth mapcapture technique.

In operation 1104, the depth data generation system may capture a seconddepth map of the surfaces of the objects included in the real-worldscene. The second depth map may be captured in operation 1104independently from the capturing of the first depth map in operation1102. The second depth map may include a second plurality of depth datapoints each representative of a different physical point included in theplurality of physical points on the surfaces of the objects included inthe real-world scene. Operation 1104 may be performed in any of the waysdescribed herein. For example, the second depth map may be captured fromthe same particular node disposed at the same particular fixed nodeposition from which the first depth map is captured in operation 1102.However, in contrast with operation 1102, operation 1104 may beperformed using a stereoscopic depth map capture technique.

In operation 1106, the depth data generation system may converge thefirst and second depth maps into a converged depth map of the surfacesof the objects included in the real-world scene. For example, theconverged depth map may include a third plurality of depth data pointseach representative of a different physical point included in theplurality of physical points on the surfaces of the objects included inthe real-world scene. Operation 1106 may be performed in any of the waysdescribed herein. For example, as illustrated in FIG. 11, operation 1106may be performed by performing operations 1108 through 1112, describedbelow.

In operation 1108, the depth data generation system may assign a firstconfidence value to a first depth data point in the first plurality ofdepth data points of the first depth map. For example, the first depthdata point may represent a particular physical point included in theplurality of physical points. Operation 1108 may be performed in any ofthe ways described herein.

In operation 1110, the depth data generation system may assign a secondconfidence value to a second depth data point in the second plurality ofdepth data points of the second depth map. In some examples, the seconddepth data point, like the first depth data point, may also representthe particular physical point. Operation 1110 may be performed in any ofthe ways described herein.

In operation 1112, the depth data generation system may generate a thirddepth data point in the third plurality of depth data points. The thirddepth data point may also represent the particular physical point.Operation 1112 may be performed in any of the ways described herein. Forexample, the third depth data point may be generated based on the firstand second confidence values and on at least one of the first depth datapoint in the first plurality of depth data points and the second depthdata point in the second plurality of depth data points.

In certain embodiments, one or more of the systems, components, and/orprocesses described herein may be implemented and/or performed by one ormore appropriately configured computing devices. To this end, one ormore of the systems and/or components described above may include or beimplemented by any computer hardware and/or computer-implementedinstructions (e.g., software) embodied on at least one non-transitorycomputer-readable medium configured to perform one or more of theprocesses described herein. In particular, system components may beimplemented on one physical computing device or may be implemented onmore than one physical computing device. Accordingly, system componentsmay include any number of computing devices, and may employ any of anumber of computer operating systems.

In certain embodiments, one or more of the processes described hereinmay be implemented at least in part as instructions embodied in anon-transitory computer-readable medium and executable by one or morecomputing devices. In general, a processor (e.g., a microprocessor)receives instructions, from a non-transitory computer-readable medium,(e.g., a memory, etc.), and executes those instructions, therebyperforming one or more processes, including one or more of the processesdescribed herein. Such instructions may be stored and/or transmittedusing any of a variety of known computer-readable media.

A computer-readable medium (also referred to as a processor-readablemedium) includes any non-transitory medium that participates inproviding data (e.g., instructions) that may be read by a computer(e.g., by a processor of a computer). Such a medium may take many forms,including, but not limited to, non-volatile media, and/or volatilemedia. Non-volatile media may include, for example, optical or magneticdisks and other persistent memory. Volatile media may include, forexample, dynamic random access memory (“DRAM”), which typicallyconstitutes a main memory. Common forms of computer-readable mediainclude, for example, a disk, hard disk, magnetic tape, any othermagnetic medium, a compact disc read-only memory (“CD-ROM”), a digitalvideo disc (“DVD”), any other optical medium, random access memory(“RAM”), programmable read-only memory (“PROM”), electrically erasableprogrammable read-only memory (“EPROM”), FLASH-EEPROM, any other memorychip or cartridge, or any other tangible medium from which a computercan read.

FIG. 12 illustrates an exemplary computing device 1200 that may bespecifically configured to perform one or more of the processesdescribed herein. As shown in FIG. 12, computing device 1200 may includea communication interface 1202, a processor 1204, a storage device 1206,and an input/output (“I/O”) module 1208 communicatively connected via acommunication infrastructure 1210. While an exemplary computing device1200 is shown in FIG. 12, the components illustrated in FIG. 12 are notintended to be limiting. Additional or alternative components may beused in other embodiments. Components of computing device 1200 shown inFIG. 12 will now be described in additional detail.

Communication interface 1202 may be configured to communicate with oneor more computing devices. Examples of communication interface 1202include, without limitation, a wired network interface (such as anetwork interface card), a wireless network interface (such as awireless network interface card), a modem, an audio/video connection,and any other suitable interface.

Processor 1204 generally represents any type or form of processing unitcapable of processing data or interpreting, executing, and/or directingexecution of one or more of the instructions, processes, and/oroperations described herein. Processor 1204 may direct execution ofoperations in accordance with one or more applications 1212 or othercomputer-executable instructions such as may be stored in storage device1206 or another computer-readable medium.

Storage device 1206 may include one or more data storage media, devices,or configurations and may employ any type, form, and combination of datastorage media and/or device. For example, storage device 1206 mayinclude, but is not limited to, a hard drive, network drive, flashdrive, magnetic disc, optical disc, RAM, dynamic RAM, other non-volatileand/or volatile data storage units, or a combination or sub-combinationthereof. Electronic data, including data described herein, may betemporarily and/or permanently stored in storage device 1206. Forexample, data representative of one or more executable applications 1212configured to direct processor 1204 to perform any of the operationsdescribed herein may be stored within storage device 1206. In someexamples, data may be arranged in one or more databases residing withinstorage device 1206.

I/O module 1208 may include one or more I/O modules configured toreceive user input and provide user output. One or more I/O modules maybe used to receive input for a single virtual reality experience. I/Omodule 1208 may include any hardware, firmware, software, or combinationthereof supportive of input and output capabilities. For example, I/Omodule 1208 may include hardware and/or software for capturing userinput, including, but not limited to, a keyboard or keypad, atouchscreen component (e.g., touchscreen display), a receiver (e.g., anRF or infrared receiver), motion sensors, and/or one or more inputbuttons.

I/O module 1208 may include one or more devices for presenting output toa user, including, but not limited to, a graphics engine, a display(e.g., a display screen), one or more output drivers (e.g., displaydrivers), one or more audio speakers, and one or more audio drivers. Incertain embodiments, I/O module 1208 is configured to provide graphicaldata to a display for presentation to a user. The graphical data may berepresentative of one or more graphical user interfaces and/or any othergraphical content as may serve a particular implementation.

In some examples, any of the facilities described herein may beimplemented by or within one or more components of computing device1200. For example, one or more applications 1212 residing within storagedevice 1206 may be configured to direct processor 1204 to perform one ormore processes or functions associated with depth map access facility102 or depth map convergence facility 104 (see FIG. 1). Likewise,storage facility 106 of system 100 may be implemented by or withinstorage device 1206.

To the extent the aforementioned embodiments collect, store, and/oremploy personal information provided by individuals, it should beunderstood that such information shall be user in accordance with allapplicable laws concerning protection of personal information.Additionally, the collection, storage, and use of such information maybe subject to consent of the individual to such activity, for example,through well known “opt-in” or “opt-out” processes as may be appropriatefor the situation and type of information. Storage and use of personalinformation may be in an appropriately secure manner reflective of thetype of information, for example, through various encryption andanonymization techniques for particularly sensitive information.

In the preceding description, various exemplary embodiments have beendescribed with reference to the accompanying drawings. It will, however,be evident that various modifications and changes may be made thereto,and additional embodiments may be implemented, without departing fromthe scope of the invention as set forth in the claims that follow. Forexample, certain features of one embodiment described herein may becombined with or substituted for features of another embodimentdescribed herein. The description and drawings are accordingly to beregarded in an illustrative rather than a restrictive sense.

What is claimed is:
 1. A method comprising: accessing, by a depth datageneration system, a first depth map of surfaces of objects included ina real-world scene, the first depth map including a first plurality ofdepth data points each representative of a different physical pointincluded in a plurality of physical points on the surfaces of theobjects included in the real-world scene; accessing, by the depth datageneration system, a second depth map of the surfaces of the objectsincluded in the real-world scene, the second depth map capturedindependently from the first depth map and including a second pluralityof depth data points each representative of a different physical pointincluded in the plurality of physical points on the surfaces of theobjects included in the real-world scene; and converging, by the depthdata generation system, the first and second depth maps into a convergeddepth map of the surfaces of the objects included in the real-worldscene, the converged depth map including a third plurality of depth datapoints each representative of a different physical point included in theplurality of physical points on the surfaces of the objects included inthe real-world scene, the converging comprising: assigning a firstconfidence value to a first depth data point in the first plurality ofdepth data points of the first depth map, the first depth data pointrepresenting a particular physical point included in the plurality ofphysical points, assigning a second confidence value to a second depthdata point in the second plurality of depth data points of the seconddepth map, the second depth data point representing the particularphysical point, and generating, based on the first and second confidencevalues and on at least one of the first depth data point in the firstplurality of depth data points and the second depth data point in thesecond plurality of depth data points, a third depth data point in thethird plurality of depth data points, the third depth data pointrepresenting the particular physical point.
 2. The method of claim 1,further comprising: generating, by the depth data generation systembased on the converged depth map, a data stream representative of adynamic volumetric model of the surfaces of the objects included in thereal-world scene, the dynamic volumetric model of the surfaces of theobjects in the real-world scene configured to be used to generatevirtual reality media content representative of the real-world scene asexperienced from a dynamically selectable viewpoint corresponding to anarbitrary location within the real-world scene, the dynamicallyselectable viewpoint selected by a user of a media player device whilethe user is experiencing the real-world scene using the media playerdevice; and providing, by the depth data generation system to the mediaplayer device and based on the data stream, the virtual reality mediacontent representative of the real-world scene.
 3. The method of claim1, wherein: the assigning of the first confidence value to the firstdepth data point is performed based on at least one of a first attributeof a particular surface that includes the particular physical point, theparticular surface belonging to a particular object in the objectsincluded in the real-world scene, and a first viewing perspective, withrespect to the particular surface that includes the particular physicalpoint, of a component of the depth data generation system capturing thefirst depth map; and the assigning of the second confidence value to thesecond depth data point is performed based on at least one of a secondattribute of the particular surface that includes the particularphysical point, and a second viewing perspective, with respect to theparticular surface that includes the particular physical point, of acomponent of the depth data generation system capturing the second depthmap.
 4. The method of claim 1, wherein the generating of the third depthdata point comprises at least one of: identifying a maximum confidencevalue from the first and second confidence values and, if the maximumconfidence value is the first confidence value, assigning the firstdepth data point to the third depth data point, and if the maximumconfidence value is the second confidence value, assigning the seconddepth data point to the third depth data point; and generating the thirddepth data point as a weighted average of the first and second depthdata points in accordance with the first and second confidence values.5. The method of claim 1, wherein: the accessing of the first depth mapincludes capturing the first depth map by way of a first depth mapcapture technique; and the accessing of the second depth map includescapturing the second depth map by way of a second depth map capturetechnique different from the first depth map capture technique.
 6. Themethod of claim 5, wherein: the depth data generation system includes aplurality of nodes each disposed at a different fixed node position in aplurality of fixed node positions with respect to the real-world scene;the capturing of the first depth map and the capturing of the seconddepth map are performed from a particular node in the plurality ofnodes; and the first and second depth map capture techniques aredifferent depth map capture techniques each selected from: a structuredlight depth map capture technique, a stereoscopic depth map capturetechnique, and a time-of-flight depth map capture technique.
 7. Themethod of claim 1, wherein: the accessing of the first depth mapincludes capturing the first depth map by way of a depth map capturetechnique; and the accessing of the second depth map includes capturingthe second depth map by way of the depth map capture technique.
 8. Themethod of claim 7, wherein: the depth data generation system includes aplurality of nodes each disposed at a different fixed node position in aplurality of fixed node positions with respect to the real-world scene;the capturing of the first depth map is performed from a first node inthe plurality of nodes; the capturing of the second depth map isperformed from a second node in the plurality of nodes, the second nodedisposed at a different fixed node position than the first node in theplurality of fixed node positions; and the depth map capture techniqueis selected from: a structured light depth map capture technique, astereoscopic depth map capture technique, and a time-of-flight depth mapcapture technique.
 9. The method of claim 1, wherein: the accessing ofthe first depth map includes converging a first plurality of antecedentdepth maps into the first depth map, the antecedent depth maps in thefirst plurality of antecedent depth maps each captured by way of a firstdepth map capture technique; and the accessing of the second depth mapincludes converging a second plurality of antecedent depth maps into thesecond depth map, the antecedent depth maps in the second plurality ofantecedent depth maps each captured by way of a second depth map capturetechnique.
 10. The method of claim 1, embodied as computer-executableinstructions on at least one non-transitory computer-readable medium.11. A method comprising: capturing, by a depth data generation system, afirst depth map of surfaces of objects included in a real-world scene,the first depth map including a first plurality of depth data pointseach representative of a different physical point included in aplurality of physical points on the surfaces of the objects included inthe real-world scene, the capturing of the first depth map performedfrom a particular node disposed at a particular fixed node position withrespect to the real-world scene and using a structured light depth mapcapture technique; capturing, by the depth data generation systemindependently from the capturing of the first depth map, a second depthmap of the surfaces of the objects included in the real-world scene, thesecond depth map including a second plurality of depth data points eachrepresentative of a different physical point included in the pluralityof physical points on the surfaces of the objects included in thereal-world scene, the capturing of the second depth map performed fromthe particular node disposed at the particular fixed node position andusing a stereoscopic depth map capture technique; converging, by thedepth data generation system, the first and second depth maps into aconverged depth map of the surfaces of the objects included in thereal-world scene, the converged depth map including a third plurality ofdepth data points each representative of a different physical pointincluded in the plurality of physical points on the surfaces of theobjects included in the real-world scene, the converging comprising:assigning a first confidence value to a first depth data point in thefirst plurality of depth data points of the first depth map, the firstdepth data point representing a particular physical point included inthe plurality of physical points, assigning a second confidence value toa second depth data point in the second plurality of depth data pointsof the second depth map, the second depth data point representing theparticular physical point, and generating, based on the first and secondconfidence values and on at least one of the first depth data point inthe first plurality of depth data points and the second depth data pointin the second plurality of depth data points, a third depth data pointin the third plurality of depth data points, the third depth data pointrepresenting the particular physical point.
 12. The method of claim 11,embodied as computer-executable instructions on at least onenon-transitory computer-readable medium.
 13. A system comprising: atleast one physical computing device that accesses a first depth map ofsurfaces of objects included in a real-world scene, the first depth mapincluding a first plurality of depth data points each representative ofa different physical point included in a plurality of physical points onthe surfaces of the objects included in the real-world scene; accesses asecond depth map of the surfaces of the objects included in thereal-world scene, the second depth map captured independently from thefirst depth map and including a second plurality of depth data pointseach representative of a different physical point included in theplurality of physical points on the surfaces of the objects included inthe real-world scene; and converges the first and second depth maps intoa converged depth map of the surfaces of the objects included in thereal-world scene, the converged depth map including a third plurality ofdepth data points each representative of a different physical pointincluded in the plurality of physical points on the surfaces of theobjects included in the real-world scene, by assigning a firstconfidence value to a first depth data point in the first plurality ofdepth data points of the first depth map, the first depth data pointrepresenting a particular physical point included in the plurality ofphysical points, assigning a second confidence value to a second depthdata point in the second plurality of depth data points of the seconddepth map, the second depth data point representing the particularphysical point, and generating, based on the first and second confidencevalues and on at least one of the first depth data point in the firstplurality of depth data points and the second depth data point in thesecond plurality of depth data points, a third depth data point in thethird plurality of depth data points, the third depth data pointrepresenting the particular physical point.
 14. The system of claim 13,wherein the at least one physical computing device further: generates,based on the converged depth map, a data stream representative of adynamic volumetric model of the surfaces of the objects included in thereal-world scene, the dynamic volumetric model of the surfaces of theobjects in the real-world scene configured to be used to generatevirtual reality media content representative of the real-world scene asexperienced from a dynamically selectable viewpoint corresponding to anarbitrary location within the real-world scene, the dynamicallyselectable viewpoint selected by a user of a media player device whilethe user is experiencing the real-world scene using the media playerdevice; and provides, to the media player device and based on the datastream, the virtual reality media content representative of thereal-world scene as experienced from the dynamically selectableviewpoint corresponding to the arbitrary location within the real-worldscene.
 15. The system of claim 13, wherein: the at least one physicalcomputing device assigns the first confidence value to the first depthdata point based on at least one of a first attribute of a particularsurface that includes the particular physical point, the particularsurface belonging to a particular object in the objects included in thereal-world scene, and a first viewing perspective, with respect to theparticular surface that includes the particular physical point, of acomponent of the system capturing the first depth map; and the at leastone physical computing device assigns the second confidence value to thesecond depth data point based on at least one of a second attribute ofthe particular surface that includes the particular physical point, anda second viewing perspective, with respect to the particular surfacethat includes the particular physical point, of a component of thesystem capturing the second depth map.
 16. The system of claim 13,wherein the at least one physical computing device generates the thirddepth data point by at least one of: identifying a maximum confidencevalue from the first and second confidence values and, if the maximumconfidence value is the first confidence value, assigning the firstdepth data point to the third depth data point, and if the maximumconfidence value is the second confidence value, assigning the seconddepth data point to the third depth data point; and generating the thirddepth data point as a weighted average of the first and second depthdata points in accordance with the first and second confidence values.17. The system of claim 13, wherein: the at least one physical computingdevice accesses the first depth map by capturing the first depth map byway of a first depth map capture technique; and the at least onephysical computing device accesses the second depth map by capturing thesecond depth map by way of a second depth map capture techniquedifferent from the first depth map capture technique.
 18. The system ofclaim 17, wherein: the at least one physical computing device isdistributed across a plurality of nodes each disposed at a differentfixed node position in a plurality of fixed node positions with respectto the real-world scene; the capturing of the first depth map and thecapturing of the second depth map are performed from a particular nodein the plurality of nodes; and the first and second depth map capturetechniques are different depth map capture techniques each selectedfrom: a structured light depth map capture technique, a stereoscopicdepth map capture technique, and a time-of-flight depth map capturetechnique.
 19. The system of claim 13, wherein: the at least onephysical computing device accesses the first depth map by capturing thefirst depth map by way of a depth map capture technique; and the atleast one physical computing device accesses the second depth map bycapturing the second depth map by way of the depth map capturetechnique.
 20. The system of claim 19, wherein: the at least onephysical computing device is distributed across a plurality of nodeseach disposed at a different fixed node position in a plurality of fixednode positions with respect to the real-world scene; the capturing ofthe first depth map is performed from a first node in the plurality ofnodes; the capturing of the second depth map is performed from a secondnode in the plurality of nodes, the second node disposed at a differentfixed node position than the first node in the plurality of fixed nodepositions; and the depth map capture technique is selected from: astructured light depth map capture technique, a stereoscopic depth mapcapture technique, and a time-of-flight depth map capture technique.